2024-03-29T11:28:54.313 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:28:54.313 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733 --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/orig.config.yaml 2024-03-29T11:28:54.350 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:28:54.440 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622733' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDn8hr+R//BuBItQCC5dVoSKBKDvgSrKDS+QH9rlJSAYWFy2nL/xzTn7JCmO18nSwnYdEASNcybaXfNAOMWEN/4= smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOFSxlvW3aKHYhb1nDKtEDx052k4LRqJy3lWvFvDkLSuD/IqfruqCmgITtVgCg5NNInCZA9x6Fe/yzLIJ4Iktqs= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:28:54.441 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:28:54.441 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:28:54.441 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:28:54.444 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:28:54.470 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:28:54.471 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:28:54.471 INFO:teuthology.packaging:ref: None 2024-03-29T11:28:54.471 INFO:teuthology.packaging:tag: None 2024-03-29T11:28:54.471 INFO:teuthology.packaging:branch: main 2024-03-29T11:28:54.471 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:28:54.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:28:54.837 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:28:54.839 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:28:54.853 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:28:54.853 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:28:54.888 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:28:54.901 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:28:54.913 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:28:54.936 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:22:16.193973', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDn8hr+R//BuBItQCC5dVoSKBKDvgSrKDS+QH9rlJSAYWFy2nL/xzTn7JCmO18nSwnYdEASNcybaXfNAOMWEN/4='} 2024-03-29T11:28:54.961 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:22:16.195403', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOFSxlvW3aKHYhb1nDKtEDx052k4LRqJy3lWvFvDkLSuD/IqfruqCmgITtVgCg5NNInCZA9x6Fe/yzLIJ4Iktqs='} 2024-03-29T11:28:54.961 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:28:55.005 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:28:55.005 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:28:55.005 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:28:55.108 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2b118760d0>, signals=[15]) 2024-03-29T11:28:55.109 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:28:55.115 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:28:55.115 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:28:55.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:28:55.222 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:28:55.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:28:55.315 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:28:55.322 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2024-03-29T11:28:55.340 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2024-03-29T11:28:55.340 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2024-03-29T11:28:55.397 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2024-03-29T11:28:55.397 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="9" 2024-03-29T11:28:55.397 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2024-03-29T11:28:55.397 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="9" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:28:55.398 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:28:55.399 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2024-03-29T11:28:55.424 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2024-03-29T11:28:55.442 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2024-03-29T11:28:55.443 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="9" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="9" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:28:55.500 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:28:55.501 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:28:55.501 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2024-03-29T11:28:55.523 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:28:55.533 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:28:55.563 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:28:55.563 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:28:55.566 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:28:55.582 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:28:55.588 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:28:55.588 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:28:55.624 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:28:55.657 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:28:55.716 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:28:55.730 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:28:55.730 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:28:55.731 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:28:55.731 DEBUG:teuthology.orchestra.run.smithi171:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:28:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:28:55.748 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-03-29T11:28:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:28:55.749 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-29T11:28:55.805 INFO:teuthology.orchestra.run.smithi171.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:28:55.805 INFO:teuthology.task.kernel:Running kernel on smithi171: 5.14.0-361.el9.x86_64 2024-03-29T11:28:55.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum install -y kernel 2024-03-29T11:28:55.806 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:28:55.807 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.14.0-361.el9.x86_64 2024-03-29T11:28:55.807 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2024-03-29T11:28:58.055 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:28:58.068 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:28:59.502 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-29T11:28:59.697 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T11:29:01.129 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - BaseOS 7.3 MB/s | 8.0 MB 00:01 2024-03-29T11:29:01.968 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-03-29T11:29:03.531 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T11:29:04.636 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-29T11:29:08.861 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-29T11:29:09.123 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.1 MB 00:02 2024-03-29T11:29:10.532 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-03-29T11:29:11.370 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-29T11:29:11.575 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-29T11:29:13.358 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T11:29:16.583 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-29T11:29:18.564 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-29T11:29:18.857 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:29:18.857 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:29:18.961 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:29:18.970 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:29:18.971 INFO:teuthology.orchestra.run.smithi171.stdout:Install 4 Packages 2024-03-29T11:29:18.972 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:29:18.972 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 76 M 2024-03-29T11:29:18.972 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 123 M 2024-03-29T11:29:18.972 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:29:19.135 INFO:teuthology.orchestra.run.smithi171.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:29:19.677 INFO:teuthology.orchestra.run.smithi171.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T11:29:20.227 INFO:teuthology.orchestra.run.smithi171.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T11:29:20.535 INFO:teuthology.orchestra.run.smithi171.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T11:29:20.538 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:29:20.538 INFO:teuthology.orchestra.run.smithi171.stdout:Total 49 MB/s | 76 MB 00:01 2024-03-29T11:29:20.539 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:29:20.905 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:29:20.905 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:29:20.905 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:29:20.905 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:29:21.002 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:29:21.010 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout:Install 4 Packages 2024-03-29T11:29:21.011 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:29:21.012 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 76 M 2024-03-29T11:29:21.012 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 123 M 2024-03-29T11:29:21.012 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:29:21.157 INFO:teuthology.orchestra.run.smithi055.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:29:21.632 INFO:teuthology.orchestra.run.smithi055.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-29T11:29:21.727 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:29:21.727 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:29:22.391 INFO:teuthology.orchestra.run.smithi055.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T11:29:22.791 INFO:teuthology.orchestra.run.smithi055.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T11:29:22.794 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:29:22.794 INFO:teuthology.orchestra.run.smithi055.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-29T11:29:22.795 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:29:23.218 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:29:23.218 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:29:24.038 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:29:24.074 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:29:24.074 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:29:24.630 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:29:24.713 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:29:26.510 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:29:27.119 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:29:27.200 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:29:27.454 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:29:27.582 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:29:30.134 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:29:30.271 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:29:32.274 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:29:32.305 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:29:35.346 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:29:35.395 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:29:37.097 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:29:40.501 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:30:06.544 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:30:06.545 INFO:teuthology.orchestra.run.smithi171.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:30:06.545 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:30:06.587 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:30:06.932 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:30:06.933 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:30:06.933 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:30:06.933 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:30:07.352 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:30:07.352 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:30:07.352 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:30:07.352 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:30:07.352 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:30:07.353 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:30:07.353 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:30:07.353 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:30:07.353 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:30:07.693 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:30:07.830 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:30:07.830 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:30:07.830 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:30:07.830 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:30:07.830 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:30:07.831 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi171.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:30:07.831 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:30:07.910 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:30:07.911 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:30:07.911 DEBUG:teuthology.orchestra.run.smithi171:> sudo rpm -qi grub2-tools 2024-03-29T11:30:07.954 INFO:teuthology.orchestra.run.smithi171.stdout:Name : grub2-tools 2024-03-29T11:30:07.954 INFO:teuthology.orchestra.run.smithi171.stdout:Epoch : 1 2024-03-29T11:30:07.954 INFO:teuthology.orchestra.run.smithi171.stdout:Version : 2.06 2024-03-29T11:30:07.954 INFO:teuthology.orchestra.run.smithi171.stdout:Release : 68.el9 2024-03-29T11:30:07.954 INFO:teuthology.orchestra.run.smithi171.stdout:Architecture: x86_64 2024-03-29T11:30:07.954 INFO:teuthology.orchestra.run.smithi171.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:30:07.954 INFO:teuthology.orchestra.run.smithi171.stdout:Group : Unspecified 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:Size : 8264254 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:License : GPLv3+ 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:Packager : builder@centos.org 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:Vendor : CentOS 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:Summary : Support tools for GRUB. 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:Description : 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:30:07.955 INFO:teuthology.orchestra.run.smithi171.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:30:07.956 INFO:teuthology.orchestra.run.smithi171.stdout:hardware devices. 2024-03-29T11:30:07.956 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:30:07.956 INFO:teuthology.orchestra.run.smithi171.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:30:07.957 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:30:07.957 INFO:teuthology.task.kernel:Updating grub on smithi171 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:30:07.957 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:30:08.556 INFO:teuthology.orchestra.run.smithi171.stderr:Generating grub configuration file ... 2024-03-29T11:30:09.995 INFO:teuthology.orchestra.run.smithi171.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:30:10.039 INFO:teuthology.orchestra.run.smithi171.stderr:done 2024-03-29T11:30:10.042 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-03-29T11:30:10.057 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.mgsz4mgLEv 2024-03-29T11:30:10.057 DEBUG:teuthology.orchestra.run.smithi171:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mgsz4mgLEv 2024-03-29T11:30:10.126 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.mgsz4mgLEv 2024-03-29T11:30:10.265 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.mgsz4mgLEv is 6KB 2024-03-29T11:30:10.314 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.mgsz4mgLEv 2024-03-29T11:30:10.330 DEBUG:teuthology.orchestra.run.smithi171:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:30:10.393 INFO:teuthology.orchestra.run.smithi171.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:30:10.393 INFO:teuthology.orchestra.run.smithi171.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:30:10.393 INFO:teuthology.orchestra.run.smithi171.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:30:10.393 INFO:teuthology.orchestra.run.smithi171.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:30:10.394 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:30:10.538 DEBUG:teuthology.orchestra.run.smithi171:> sudo shutdown -r now 2024-03-29T11:30:11.345 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:30:11.345 INFO:teuthology.orchestra.run.smithi055.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:30:11.345 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:30:11.390 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:30:11.737 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:30:11.737 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:30:11.737 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:30:11.737 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:30:12.144 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:30:12.145 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:30:12.145 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:30:12.145 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:30:12.145 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:30:12.145 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:30:12.145 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:30:12.145 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:30:12.145 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:30:12.477 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:30:12.581 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:30:12.582 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:30:12.582 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:30:12.582 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:30:12.582 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:30:12.582 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:30:12.582 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:30:12.582 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:30:12.666 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:30:12.666 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:30:12.666 DEBUG:teuthology.orchestra.run.smithi055:> sudo rpm -qi grub2-tools 2024-03-29T11:30:12.704 INFO:teuthology.orchestra.run.smithi055.stdout:Name : grub2-tools 2024-03-29T11:30:12.704 INFO:teuthology.orchestra.run.smithi055.stdout:Epoch : 1 2024-03-29T11:30:12.704 INFO:teuthology.orchestra.run.smithi055.stdout:Version : 2.06 2024-03-29T11:30:12.704 INFO:teuthology.orchestra.run.smithi055.stdout:Release : 68.el9 2024-03-29T11:30:12.704 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: x86_64 2024-03-29T11:30:12.704 INFO:teuthology.orchestra.run.smithi055.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:30:12.704 INFO:teuthology.orchestra.run.smithi055.stdout:Group : Unspecified 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:Size : 8264254 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:License : GPLv3+ 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:Packager : builder@centos.org 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:Vendor : CentOS 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:Summary : Support tools for GRUB. 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:Description : 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:30:12.705 INFO:teuthology.orchestra.run.smithi055.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:30:12.706 INFO:teuthology.orchestra.run.smithi055.stdout:hardware devices. 2024-03-29T11:30:12.706 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:30:12.706 INFO:teuthology.orchestra.run.smithi055.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:30:12.707 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:30:12.707 INFO:teuthology.task.kernel:Updating grub on smithi055 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:30:12.707 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:30:13.388 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2024-03-29T11:30:14.980 INFO:teuthology.orchestra.run.smithi055.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:30:15.021 INFO:teuthology.orchestra.run.smithi055.stderr:done 2024-03-29T11:30:15.034 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-29T11:30:15.068 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.rqWCB4vKQo 2024-03-29T11:30:15.068 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rqWCB4vKQo 2024-03-29T11:30:15.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.rqWCB4vKQo 2024-03-29T11:30:15.283 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.rqWCB4vKQo is 6KB 2024-03-29T11:30:15.332 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.rqWCB4vKQo 2024-03-29T11:30:15.347 DEBUG:teuthology.orchestra.run.smithi055:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:30:15.415 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:30:15.416 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:30:15.416 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:30:15.416 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:30:15.417 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:30:15.597 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2024-03-29T11:30:40.566 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:30:40.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-29T11:30:40.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:45.602 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:30:45.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-29T11:30:45.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:58.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-03-29T11:31:04.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2024-03-29T11:31:07.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-29T11:31:07.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:31:11.042 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-03-29T11:31:13.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-29T11:31:13.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:31:23.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-29T11:31:23.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:31:41.471 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-03-29T11:31:56.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-29T11:31:56.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:31:56.990 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-03-29T11:31:57.388 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-29T11:31:57.388 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:31:57.388 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-03-29T11:31:57.403 INFO:teuthology.orchestra.run.smithi171.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:31:57.404 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:31:57.404 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:31:57.404 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:31:58.405 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:31:58.406 DEBUG:teuthology.orchestra.run.smithi171:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:31:58.473 INFO:teuthology.orchestra.run.smithi171.stdout:ttyS1 2024-03-29T11:31:58.511 DEBUG:teuthology.parallel:result is None 2024-03-29T11:32:13.100 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:32:25.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-29T11:32:25.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:32:25.482 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-29T11:32:25.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-29T11:32:25.908 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:32:25.909 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-29T11:32:25.925 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:32:25.926 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:32:25.926 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:32:25.926 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:32:26.927 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:32:26.928 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:32:27.011 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2024-03-29T11:32:27.051 DEBUG:teuthology.parallel:result is None 2024-03-29T11:32:27.051 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:32:27.060 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:32:27.061 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:32:27.063 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:32:27.083 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:32:27.113 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:32:27.146 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:32:27.146 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:27.149 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:27.191 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:32:27.206 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:32:27.206 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:32:27.212 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:32:27.253 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:32:27.266 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:32:27.267 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:32:27.281 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:32:27.283 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:32:27.296 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:32:27.296 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:32:27.312 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:32:27.353 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:32:27.372 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:32:27.372 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:32:27.387 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:32:27.410 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:32:27.481 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:32:27.532 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:32:27.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:32:27.660 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:32:27.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:32:27.728 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:32:27.773 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:32:27.830 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:32:27.914 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:32:27.959 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:32:27.959 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:32:28.029 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2024-03-29T11:32:28.031 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2024-03-29T11:32:28.073 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:28.142 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:28.408 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:32:28.419 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:32:28.419 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:32:28.475 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:32:28.509 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2024-03-29T11:32:28.568 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2024-03-29T11:32:28.570 DEBUG:teuthology.orchestra.run.smithi171:> sudo service auditd rotate 2024-03-29T11:32:28.637 INFO:teuthology.orchestra.run.smithi171.stdout:Rotating logs: 2024-03-29T11:32:28.639 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:32:28.639 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2024-03-29T11:32:28.670 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2024-03-29T11:32:28.671 DEBUG:teuthology.orchestra.run.smithi171:> /usr/sbin/getenforce 2024-03-29T11:32:28.691 INFO:teuthology.orchestra.run.smithi171.stdout:Permissive 2024-03-29T11:32:28.691 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi171.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:32:28.692 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:28.739 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:28.770 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:32:28.770 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:32:28.815 DEBUG:teuthology.orchestra.run.smithi171:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:32:28.849 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:32:28.887 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:32:28.896 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:32:29.067 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:32:29.084 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:32:29.085 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi171.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:36:48.895 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi171.front.sepia.ceph.com')] 2024-03-29T11:36:48.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-29T11:36:48.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:36:48.987 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-29T11:36:49.068 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-29T11:36:49.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-29T11:36:49.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:36:49.153 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-03-29T11:36:49.228 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-29T11:36:49.228 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:36:49.240 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:36:49.240 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:36:49.240 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:36:49.243 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:36:49.243 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:36:49.274 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:36:49.291 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:36:49.298 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:36:49.314 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:36:49.321 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2024-03-29T11:36:49.335 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2024-03-29T11:36:49.343 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: ntpd: command not found 2024-03-29T11:36:49.352 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:36:49.357 INFO:teuthology.orchestra.run.smithi171.stdout:506 Cannot talk to daemon 2024-03-29T11:36:49.368 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:36:49.374 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:36:49.390 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:36:49.422 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:36:49.426 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:36:49.426 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-03-29T11:36:49.426 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:49.426 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:49.426 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:49.426 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:49.442 INFO:teuthology.orchestra.run.smithi171.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:36:49.446 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:36:49.446 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-03-29T11:36:49.446 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:49.446 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:49.446 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:49.446 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:49.447 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:36:49.457 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:36:49.457 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:36:49.457 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:36:49.457 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:36:49.463 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:36:49.464 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:36:49.464 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:36:49.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:36:49.466 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:36:49.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:36:49.684 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:36:49.684 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:36:49.693 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:36:49.693 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:36:49.794 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:36:49.794 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:36:49.794 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:36:49.799 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:36:49.799 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:36:49.799 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:36:49.822 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:36:49.823 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:36:49.826 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:36:49.827 DEBUG:teuthology.orchestra.run.smithi171:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:36:49.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:36:49.894 DEBUG:teuthology.orchestra.run.smithi171:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:36:49.970 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:36:49.974 DEBUG:teuthology.orchestra.run.smithi171:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:36:50.036 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2024-03-29T11:36:50.038 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2024-03-29T11:36:50.041 INFO:teuthology.orchestra.run.smithi171.stdout:check_obsoletes = 1 2024-03-29T11:36:50.043 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum clean all 2024-03-29T11:36:50.277 INFO:teuthology.orchestra.run.smithi055.stdout:74 files removed 2024-03-29T11:36:50.281 INFO:teuthology.orchestra.run.smithi171.stdout:74 files removed 2024-03-29T11:36:50.304 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw 2024-03-29T11:36:50.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-radosgw 2024-03-29T11:36:50.999 INFO:teuthology.orchestra.run.smithi171.stdout:ceph packages for x86_64 211 kB/s | 82 kB 00:00 2024-03-29T11:36:51.061 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 181 kB/s | 82 kB 00:00 2024-03-29T11:36:51.357 INFO:teuthology.orchestra.run.smithi171.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-29T11:36:51.495 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-29T11:36:51.774 INFO:teuthology.orchestra.run.smithi171.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-29T11:36:51.846 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-29T11:36:52.193 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:36:52.257 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:36:53.611 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-29T11:36:53.674 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-29T11:36:54.609 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T11:36:56.581 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-03-29T11:36:58.749 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-29T11:37:03.220 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.1 MB 00:00 2024-03-29T11:37:04.660 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 16 kB 00:00 2024-03-29T11:37:05.854 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-29T11:37:08.792 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-03-29T11:37:10.984 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T11:37:13.718 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.1 MB 00:01 2024-03-29T11:37:13.740 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:37:13.760 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:37:13.760 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:37:13.760 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:37:13.760 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:37:13.761 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:37:13.761 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2024-03-29T11:37:13.761 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:37:13.761 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:37:13.761 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:37:13.762 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options 2024-03-29T11:37:13.762 INFO:teuthology.orchestra.run.smithi055.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:37:13.762 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:37:13.762 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:37:13.762 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:37:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:37:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:37:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:37:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:37:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:37:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:37:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:37:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:37:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:37:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:37:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:37:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:37:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:37:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse 2024-03-29T11:37:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:37:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:37:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:37:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout:Installing weak dependencies: 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout:Install 30 Packages 2024-03-29T11:37:13.767 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 2 Packages 2024-03-29T11:37:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:13.768 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 58 M 2024-03-29T11:37:13.768 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:37:14.547 INFO:teuthology.orchestra.run.smithi055.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 8.4 MB/s | 5.3 MB 00:00 2024-03-29T11:37:14.589 INFO:teuthology.orchestra.run.smithi055.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 603 kB/s | 25 kB 00:00 2024-03-29T11:37:14.697 INFO:teuthology.orchestra.run.smithi055.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.7 MB/s | 742 kB 00:00 2024-03-29T11:37:14.790 INFO:teuthology.orchestra.run.smithi055.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.3 MB/s | 498 kB 00:00 2024-03-29T11:37:15.306 INFO:teuthology.orchestra.run.smithi055.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.9 MB/s | 5.1 MB 00:00 2024-03-29T11:37:15.344 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-29T11:37:15.348 INFO:teuthology.orchestra.run.smithi055.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T11:37:15.466 INFO:teuthology.orchestra.run.smithi055.stdout:(7/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 6.3 MB/s | 9.7 MB 00:01 2024-03-29T11:37:15.491 INFO:teuthology.orchestra.run.smithi055.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 925 kB/s | 131 kB 00:00 2024-03-29T11:37:15.549 INFO:teuthology.orchestra.run.smithi055.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 5.4 MB/s | 320 kB 00:00 2024-03-29T11:37:15.575 INFO:teuthology.orchestra.run.smithi055.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 1.5 MB/s | 162 kB 00:00 2024-03-29T11:37:15.608 INFO:teuthology.orchestra.run.smithi055.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T11:37:15.667 INFO:teuthology.orchestra.run.smithi055.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-29T11:37:15.718 INFO:teuthology.orchestra.run.smithi055.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 971 kB/s | 106 kB 00:00 2024-03-29T11:37:15.743 INFO:teuthology.orchestra.run.smithi055.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-29T11:37:15.768 INFO:teuthology.orchestra.run.smithi055.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-29T11:37:15.810 INFO:teuthology.orchestra.run.smithi055.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T11:37:15.835 INFO:teuthology.orchestra.run.smithi055.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-29T11:37:15.861 INFO:teuthology.orchestra.run.smithi055.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-29T11:37:15.886 INFO:teuthology.orchestra.run.smithi055.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:37:15.911 INFO:teuthology.orchestra.run.smithi055.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-29T11:37:15.938 INFO:teuthology.orchestra.run.smithi055.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 922 kB/s | 46 kB 00:00 2024-03-29T11:37:16.137 INFO:teuthology.orchestra.run.smithi055.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 8.9 MB/s | 20 MB 00:02 2024-03-29T11:37:16.345 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-29T11:37:16.429 INFO:teuthology.orchestra.run.smithi055.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 108 kB/s | 26 kB 00:00 2024-03-29T11:37:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 552 kB/s | 309 kB 00:00 2024-03-29T11:37:16.505 INFO:teuthology.orchestra.run.smithi055.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 648 kB/s | 49 kB 00:00 2024-03-29T11:37:16.547 INFO:teuthology.orchestra.run.smithi055.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 896 kB/s | 67 kB 00:00 2024-03-29T11:37:16.647 INFO:teuthology.orchestra.run.smithi055.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-29T11:37:16.673 INFO:teuthology.orchestra.run.smithi055.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-29T11:37:16.748 INFO:teuthology.orchestra.run.smithi055.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-03-29T11:37:16.807 INFO:teuthology.orchestra.run.smithi055.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-29T11:37:17.015 INFO:teuthology.orchestra.run.smithi055.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 9.9 MB/s | 3.4 MB 00:00 2024-03-29T11:37:17.273 INFO:teuthology.orchestra.run.smithi055.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 6.0 MB/s | 3.1 MB 00:00 2024-03-29T11:37:17.280 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:37:17.280 INFO:teuthology.orchestra.run.smithi055.stdout:Total 16 MB/s | 58 MB 00:03 2024-03-29T11:37:17.281 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:37:17.322 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:37:17.322 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:37:17.821 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:37:17.821 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:37:19.339 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:37:19.411 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:37:19.437 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:37:19.466 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:37:19.494 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:37:19.511 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:37:19.694 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:37:19.751 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:37:20.149 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:37:20.190 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:37:20.282 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:37:20.372 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:37:20.399 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:37:20.448 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:37:20.483 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:37:20.528 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:37:20.597 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:37:20.619 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:37:20.683 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:37:20.718 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:37:20.830 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:37:20.861 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:37:20.903 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:37:20.942 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:37:20.971 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:37:21.004 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:37:21.321 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-29T11:37:21.345 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:37:21.376 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:37:21.450 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:37:21.481 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:37:21.523 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:37:21.998 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:37:22.029 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:37:22.101 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:37:22.171 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:37:22.198 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:37:23.966 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrading: 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options 2024-03-29T11:37:23.983 INFO:teuthology.orchestra.run.smithi171.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:37:23.984 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:37:23.985 INFO:teuthology.orchestra.run.smithi171.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:37:23.986 INFO:teuthology.orchestra.run.smithi171.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:37:23.986 INFO:teuthology.orchestra.run.smithi171.stdout:Installing weak dependencies: 2024-03-29T11:37:23.986 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:37:23.986 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:37:23.986 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:37:23.986 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:37:23.986 INFO:teuthology.orchestra.run.smithi171.stdout:Install 30 Packages 2024-03-29T11:37:23.986 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrade 2 Packages 2024-03-29T11:37:23.986 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:37:23.987 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 58 M 2024-03-29T11:37:23.987 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:37:24.434 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:37:24.479 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:37:25.054 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:37:25.091 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:37:25.172 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:37:25.173 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:37:25.173 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:37:25.173 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:25.226 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:37:25.471 INFO:teuthology.orchestra.run.smithi171.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.5 MB/s | 9.7 MB 00:01 2024-03-29T11:37:25.513 INFO:teuthology.orchestra.run.smithi171.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 608 kB/s | 25 kB 00:00 2024-03-29T11:37:25.663 INFO:teuthology.orchestra.run.smithi171.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.8 MB/s | 742 kB 00:00 2024-03-29T11:37:25.772 INFO:teuthology.orchestra.run.smithi171.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.5 MB/s | 498 kB 00:00 2024-03-29T11:37:26.122 INFO:teuthology.orchestra.run.smithi171.stdout:(5/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 10 MB/s | 20 MB 00:01 2024-03-29T11:37:26.206 INFO:teuthology.orchestra.run.smithi171.stdout:(6/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.6 MB/s | 5.3 MB 00:02 2024-03-29T11:37:26.231 INFO:teuthology.orchestra.run.smithi171.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 416 kB/s | 45 kB 00:00 2024-03-29T11:37:26.281 INFO:teuthology.orchestra.run.smithi171.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-29T11:37:26.357 INFO:teuthology.orchestra.run.smithi171.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.7 MB/s | 5.1 MB 00:00 2024-03-29T11:37:26.382 INFO:teuthology.orchestra.run.smithi171.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 1.1 MB/s | 162 kB 00:00 2024-03-29T11:37:26.407 INFO:teuthology.orchestra.run.smithi171.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.5 MB/s | 320 kB 00:00 2024-03-29T11:37:26.432 INFO:teuthology.orchestra.run.smithi171.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T11:37:26.499 INFO:teuthology.orchestra.run.smithi171.stdout:(13/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.5 MB/s | 298 kB 00:00 2024-03-29T11:37:26.566 INFO:teuthology.orchestra.run.smithi171.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 667 kB/s | 106 kB 00:00 2024-03-29T11:37:26.591 INFO:teuthology.orchestra.run.smithi171.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 674 kB/s | 44 kB 00:00 2024-03-29T11:37:26.650 INFO:teuthology.orchestra.run.smithi171.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 831 kB/s | 159 kB 00:00 2024-03-29T11:37:26.775 INFO:teuthology.orchestra.run.smithi171.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 661 kB 00:00 2024-03-29T11:37:26.801 INFO:teuthology.orchestra.run.smithi171.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 282 kB 00:00 2024-03-29T11:37:26.817 INFO:teuthology.orchestra.run.smithi171.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 247 kB/s | 41 kB 00:00 2024-03-29T11:37:26.843 INFO:teuthology.orchestra.run.smithi171.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T11:37:26.859 INFO:teuthology.orchestra.run.smithi171.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 596 kB/s | 35 kB 00:00 2024-03-29T11:37:26.901 INFO:teuthology.orchestra.run.smithi171.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 556 kB/s | 46 kB 00:00 2024-03-29T11:37:30.393 INFO:teuthology.orchestra.run.smithi171.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 87 kB/s | 309 kB 00:03 2024-03-29T11:37:30.493 INFO:teuthology.orchestra.run.smithi171.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-29T11:37:30.552 INFO:teuthology.orchestra.run.smithi171.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T11:37:30.819 INFO:teuthology.orchestra.run.smithi171.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-29T11:37:30.919 INFO:teuthology.orchestra.run.smithi171.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T11:37:31.236 INFO:teuthology.orchestra.run.smithi171.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-03-29T11:37:31.928 INFO:teuthology.orchestra.run.smithi171.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.9 MB/s | 3.4 MB 00:00 2024-03-29T11:37:32.328 INFO:teuthology.orchestra.run.smithi171.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.8 MB/s | 3.1 MB 00:00 2024-03-29T11:37:35.425 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:37:35.425 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2024-03-29T11:37:35.426 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2024-03-29T11:37:35.426 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2024-03-29T11:37:35.426 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2024-03-29T11:37:35.426 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2024-03-29T11:37:35.426 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /root 2024-03-29T11:37:35.426 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2024-03-29T11:37:35.426 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:36.352 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:37:36.376 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:37:36.377 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:37:36.377 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:37:36.377 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:37:36.377 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:37:36.377 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:36.379 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:37:36.463 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:37:36.463 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:37:37.575 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:37:37.575 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:37:37.575 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:37:37.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:37:37.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:37:37.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:37:37.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:37:38.580 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:37:38.580 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:38.580 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2024-03-29T11:37:38.580 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:37:38.581 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:37:38.582 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:37:38.583 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:37:38.583 INFO:teuthology.orchestra.run.smithi055.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:37:38.583 INFO:teuthology.orchestra.run.smithi055.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:37:38.583 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:38.583 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:37:38.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-test 2024-03-29T11:37:39.497 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:37:39.654 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:37:39.654 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:37:39.654 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2024-03-29T11:37:39.655 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:39.656 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 48 M 2024-03-29T11:37:39.656 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 204 M 2024-03-29T11:37:39.656 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:37:39.891 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:37:39.957 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 689 kB/s | 163 kB 00:00 2024-03-29T11:37:39.982 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 300 kB 00:00 2024-03-29T11:37:39.999 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T11:37:40.266 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 660 kB/s | 187 kB 00:00 2024-03-29T11:37:43.667 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 47 MB 00:03 2024-03-29T11:37:43.670 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:37:43.670 INFO:teuthology.orchestra.run.smithi055.stdout:Total 12 MB/s | 48 MB 00:04 2024-03-29T11:37:43.684 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:37:43.707 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:37:43.707 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:37:44.109 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:37:44.109 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:37:44.700 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:37:44.769 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:37:44.809 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:37:44.850 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:37:44.893 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:37:44.920 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:37:49.656 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:37:49.739 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:37:52.147 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:37:52.148 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:37:52.148 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:37:52.148 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:37:52.148 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:37:52.148 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:37:52.514 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:37:52.514 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:52.514 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:37:52.514 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:52.514 INFO:teuthology.orchestra.run.smithi055.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:37:52.514 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:37:52.514 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:37:52.514 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:37:52.515 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:37:52.515 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:52.515 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:37:52.796 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph 2024-03-29T11:37:53.365 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:43 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:37:53.525 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:37:53.527 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2024-03-29T11:37:53.528 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:37:53.528 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2024-03-29T11:37:53.528 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:37:53.528 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:37:53.528 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:37:53.528 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:37:53.528 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:37:53.529 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:37:53.529 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:37:53.529 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:37:53.529 INFO:teuthology.orchestra.run.smithi055.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:37:53.529 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:37:53.529 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:37:53.529 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:37:53.530 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:53.531 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:37:53.532 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2024-03-29T11:37:53.532 INFO:teuthology.orchestra.run.smithi055.stdout:Install 36 Packages 2024-03-29T11:37:53.532 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:37:53.532 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 30 M 2024-03-29T11:37:53.532 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 107 M 2024-03-29T11:37:53.532 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:37:54.379 INFO:teuthology.orchestra.run.smithi055.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-03-29T11:37:54.812 INFO:teuthology.orchestra.run.smithi055.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-03-29T11:37:54.854 INFO:teuthology.orchestra.run.smithi055.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-03-29T11:37:54.938 INFO:teuthology.orchestra.run.smithi055.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.9 MB/s | 2.1 MB 00:00 2024-03-29T11:37:54.963 INFO:teuthology.orchestra.run.smithi055.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.2 MB/s | 247 kB 00:00 2024-03-29T11:37:55.096 INFO:teuthology.orchestra.run.smithi055.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:37:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T11:37:55.147 INFO:teuthology.orchestra.run.smithi055.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-03-29T11:37:55.172 INFO:teuthology.orchestra.run.smithi055.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 946 kB/s | 47 kB 00:00 2024-03-29T11:37:55.197 INFO:teuthology.orchestra.run.smithi055.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-29T11:37:55.223 INFO:teuthology.orchestra.run.smithi055.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:37:55.273 INFO:teuthology.orchestra.run.smithi055.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:37:55.298 INFO:teuthology.orchestra.run.smithi055.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T11:37:55.315 INFO:teuthology.orchestra.run.smithi055.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T11:37:55.341 INFO:teuthology.orchestra.run.smithi055.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T11:37:55.408 INFO:teuthology.orchestra.run.smithi055.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 335 kB/s | 22 kB 00:00 2024-03-29T11:37:55.450 INFO:teuthology.orchestra.run.smithi055.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 934 kB/s | 125 kB 00:00 2024-03-29T11:37:55.634 INFO:teuthology.orchestra.run.smithi055.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 20 MB/s | 17 MB 00:00 2024-03-29T11:37:56.000 INFO:teuthology.orchestra.run.smithi055.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 79 kB/s | 43 kB 00:00 2024-03-29T11:37:56.101 INFO:teuthology.orchestra.run.smithi055.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 218 kB/s | 151 kB 00:00 2024-03-29T11:37:56.217 INFO:teuthology.orchestra.run.smithi055.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 92 kB/s | 11 kB 00:00 2024-03-29T11:37:56.259 INFO:teuthology.orchestra.run.smithi055.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 300 kB/s | 172 kB 00:00 2024-03-29T11:37:56.318 INFO:teuthology.orchestra.run.smithi055.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-29T11:37:56.353 INFO:teuthology.orchestra.run.smithi055.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-29T11:37:56.377 INFO:teuthology.orchestra.run.smithi055.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 197 kB/s | 23 kB 00:00 2024-03-29T11:37:56.419 INFO:teuthology.orchestra.run.smithi055.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-03-29T11:37:56.452 INFO:teuthology.orchestra.run.smithi055.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 198 kB/s | 20 kB 00:00 2024-03-29T11:37:56.494 INFO:teuthology.orchestra.run.smithi055.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2024-03-29T11:37:56.545 INFO:teuthology.orchestra.run.smithi055.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 628 kB/s | 79 kB 00:00 2024-03-29T11:37:56.595 INFO:teuthology.orchestra.run.smithi055.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-29T11:37:56.620 INFO:teuthology.orchestra.run.smithi055.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2024-03-29T11:37:56.670 INFO:teuthology.orchestra.run.smithi055.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 717 kB/s | 90 kB 00:00 2024-03-29T11:37:56.695 INFO:teuthology.orchestra.run.smithi055.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-29T11:37:56.796 INFO:teuthology.orchestra.run.smithi055.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-29T11:37:56.821 INFO:teuthology.orchestra.run.smithi055.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-03-29T11:37:56.930 INFO:teuthology.orchestra.run.smithi055.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-03-29T11:37:56.936 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:37:56.936 INFO:teuthology.orchestra.run.smithi055.stdout:Total 8.7 MB/s | 30 MB 00:03 2024-03-29T11:37:56.961 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:37:57.008 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:37:57.008 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:37:57.427 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:37:57.427 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:37:58.789 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:37:58.849 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:37:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:37:58.910 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:37:58.984 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:37:59.018 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:37:59.050 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:37:59.148 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:37:59.224 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:37:59.270 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:37:59.345 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:37:59.397 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:37:59.442 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:37:59.501 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:37:59.564 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:37:59.763 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:37:59.805 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:37:59.851 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:37:59.886 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:37:59.916 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:37:59.995 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:38:00.034 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:38:00.068 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:38:00.108 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:38:00.143 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:38:00.178 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:38:00.273 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:38:00.317 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:38:00.393 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:38:00.537 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:38:00.665 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:38:00.833 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:38:00.871 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:38:00.897 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:38:00.897 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:38:00.897 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:38:00.897 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:38:00.897 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:38:00.897 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:02.422 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:38:02.450 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:38:02.450 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:38:02.450 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:38:02.450 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:38:02.450 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:38:02.450 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:02.915 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:38:02.938 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:38:02.938 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:38:02.938 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:38:02.938 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:38:02.938 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:38:02.938 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:03.156 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:38:03.180 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:38:03.180 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:38:03.180 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:38:03.180 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:38:03.180 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:38:03.180 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:03.249 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:38:04.957 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:38:04.957 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:38:04.957 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:38:04.957 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:38:04.958 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:38:04.959 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:38:04.960 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:38:04.960 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:38:04.960 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:38:05.894 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:38:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:38:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:05.897 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:06.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-base 2024-03-29T11:38:06.829 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:06.930 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:38:06.985 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:06.986 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-29T11:38:06.986 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:07.061 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install cephadm 2024-03-29T11:38:07.628 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:07.783 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:07.784 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:07.784 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:38:07.784 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:07.784 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:07.784 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:38:07.784 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:07.784 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:07.785 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:07.785 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-29T11:38:07.785 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:07.785 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 747 k 2024-03-29T11:38:07.785 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 754 k 2024-03-29T11:38:07.785 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:08.041 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.9 MB/s | 747 kB 00:00 2024-03-29T11:38:08.042 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:08.042 INFO:teuthology.orchestra.run.smithi055.stdout:Total 2.8 MB/s | 747 kB 00:00 2024-03-29T11:38:08.042 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:08.047 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:08.047 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:08.053 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:08.053 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:08.129 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:08.483 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:38:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:38:08.940 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:38:09.227 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:38:09.227 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:09.227 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:09.227 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:38:09.227 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:09.227 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:09.420 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:38:09.989 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:10.145 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-29T11:38:10.146 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:10.147 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 145 k 2024-03-29T11:38:10.147 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 436 k 2024-03-29T11:38:10.147 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:10.381 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 623 kB/s | 145 kB 00:00 2024-03-29T11:38:10.381 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:10.381 INFO:teuthology.orchestra.run.smithi055.stdout:Total 618 kB/s | 145 kB 00:00 2024-03-29T11:38:10.382 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:10.390 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:10.390 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:10.445 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:10.445 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:10.587 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:10.665 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:38:10.688 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:38:10.688 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:38:10.689 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:38:10.689 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:11.631 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:38:11.632 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:11.632 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:11.632 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:38:11.632 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:11.632 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:11.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr 2024-03-29T11:38:12.446 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:12.547 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:38:12.602 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:12.603 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-29T11:38:12.604 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:12.674 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:38:13.244 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:13.402 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:13.403 INFO:teuthology.orchestra.run.smithi055.stdout:====================================================================================== 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout:====================================================================================== 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:38:13.404 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:38:13.405 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:38:13.405 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:38:13.405 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:38:13.405 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:38:13.405 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:13.405 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:13.405 INFO:teuthology.orchestra.run.smithi055.stdout:====================================================================================== 2024-03-29T11:38:13.405 INFO:teuthology.orchestra.run.smithi055.stdout:Install 12 Packages 2024-03-29T11:38:13.405 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:13.406 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 6.9 M 2024-03-29T11:38:13.406 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 26 M 2024-03-29T11:38:13.406 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:14.026 INFO:teuthology.orchestra.run.smithi055.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 95 kB/s | 31 kB 00:00 2024-03-29T11:38:14.052 INFO:teuthology.orchestra.run.smithi055.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 43 kB/s | 15 kB 00:00 2024-03-29T11:38:14.169 INFO:teuthology.orchestra.run.smithi055.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.8 MB/s | 1.8 MB 00:00 2024-03-29T11:38:14.244 INFO:teuthology.orchestra.run.smithi055.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-29T11:38:14.303 INFO:teuthology.orchestra.run.smithi055.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.3 MB/s | 863 kB 00:00 2024-03-29T11:38:14.361 INFO:teuthology.orchestra.run.smithi055.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-03-29T11:38:14.562 INFO:teuthology.orchestra.run.smithi055.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-03-29T11:38:14.720 INFO:teuthology.orchestra.run.smithi055.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:38:14.753 INFO:teuthology.orchestra.run.smithi055.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 754 kB/s | 144 kB 00:00 2024-03-29T11:38:14.796 INFO:teuthology.orchestra.run.smithi055.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 408 kB/s | 31 kB 00:00 2024-03-29T11:38:14.829 INFO:teuthology.orchestra.run.smithi055.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-29T11:38:14.921 INFO:teuthology.orchestra.run.smithi055.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.6 MB/s | 2.0 MB 00:00 2024-03-29T11:38:14.926 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:14.927 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.5 MB/s | 6.9 MB 00:01 2024-03-29T11:38:14.945 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:14.963 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:14.963 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:15.147 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:15.148 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:15.791 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:15.872 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:38:15.915 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:38:15.963 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:38:16.000 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:38:16.108 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:38:16.241 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:38:16.425 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:38:16.473 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:38:16.500 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:38:16.532 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:38:16.787 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:38:16.824 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:38:17.635 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:38:17.635 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:38:17.635 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:38:17.635 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:38:17.636 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:38:17.636 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:38:17.636 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:38:17.637 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:38:17.637 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:38:17.637 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:38:17.637 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:38:17.637 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:38:17.999 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:38:18.000 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:38:18.000 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:38:18.000 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:18.000 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:18.244 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:38:18.816 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:18.976 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:18.977 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:38:18.978 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:38:18.979 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:38:18.979 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:38:18.979 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:38:18.979 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:18.979 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:18.979 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-03-29T11:38:18.979 INFO:teuthology.orchestra.run.smithi055.stdout:Install 12 Packages 2024-03-29T11:38:18.979 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:18.979 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 37 M 2024-03-29T11:38:18.980 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 209 M 2024-03-29T11:38:18.980 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:19.394 INFO:teuthology.orchestra.run.smithi055.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 224 kB/s | 32 kB 00:00 2024-03-29T11:38:19.420 INFO:teuthology.orchestra.run.smithi055.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 724 kB/s | 17 kB 00:00 2024-03-29T11:38:19.462 INFO:teuthology.orchestra.run.smithi055.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 829 kB/s | 34 kB 00:00 2024-03-29T11:38:19.529 INFO:teuthology.orchestra.run.smithi055.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:38:19.821 INFO:teuthology.orchestra.run.smithi055.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-03-29T11:38:19.887 INFO:teuthology.orchestra.run.smithi055.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-03-29T11:38:19.913 INFO:teuthology.orchestra.run.smithi055.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:38:20.005 INFO:teuthology.orchestra.run.smithi055.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-14 9.8 MB/s | 7.4 MB 00:00 2024-03-29T11:38:20.147 INFO:teuthology.orchestra.run.smithi055.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-29T11:38:20.188 INFO:teuthology.orchestra.run.smithi055.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-03-29T11:38:20.397 INFO:teuthology.orchestra.run.smithi055.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-29T11:38:20.939 INFO:teuthology.orchestra.run.smithi055.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 311 kB/s | 245 kB 00:00 2024-03-29T11:38:20.941 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:20.942 INFO:teuthology.orchestra.run.smithi055.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-29T11:38:20.985 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:21.013 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:21.013 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:21.395 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:21.395 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:22.142 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:22.221 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:38:22.252 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:38:22.349 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:38:22.389 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:38:22.572 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:38:22.613 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:38:22.696 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:38:23.406 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:38:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:38:24.844 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:38:25.945 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:38:25.971 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:38:28.507 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:38:28.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:38:28.882 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:38:28.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:38:28.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:38:28.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:38:28.884 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:28.884 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:29.269 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:38:29.844 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:30.003 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:30.004 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-29T11:38:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:38:30.004 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-29T11:38:30.004 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:38:30.004 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:38:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:38:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:38:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout:Install 13 Packages 2024-03-29T11:38:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:30.006 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.0 M 2024-03-29T11:38:30.006 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 26 M 2024-03-29T11:38:30.006 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:30.337 INFO:teuthology.orchestra.run.smithi055.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 173 kB/s | 24 kB 00:00 2024-03-29T11:38:30.362 INFO:teuthology.orchestra.run.smithi055.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-03-29T11:38:30.404 INFO:teuthology.orchestra.run.smithi055.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:38:30.430 INFO:teuthology.orchestra.run.smithi055.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:38:30.455 INFO:teuthology.orchestra.run.smithi055.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:38:30.480 INFO:teuthology.orchestra.run.smithi055.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-29T11:38:30.505 INFO:teuthology.orchestra.run.smithi055.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 169 kB/s | 52 kB 00:00 2024-03-29T11:38:30.864 INFO:teuthology.orchestra.run.smithi055.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 39 kB/s | 15 kB 00:00 2024-03-29T11:38:30.923 INFO:teuthology.orchestra.run.smithi055.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 69 kB/s | 32 kB 00:00 2024-03-29T11:38:31.131 INFO:teuthology.orchestra.run.smithi055.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-03-29T11:38:31.190 INFO:teuthology.orchestra.run.smithi055.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 296 kB/s | 203 kB 00:00 2024-03-29T11:38:31.290 INFO:teuthology.orchestra.run.smithi055.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-03-29T11:38:31.474 INFO:teuthology.orchestra.run.smithi055.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-29T11:38:31.479 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:31.479 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-03-29T11:38:31.480 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:31.496 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:31.496 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:31.601 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:31.601 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:32.132 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:32.304 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:38:32.359 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:38:32.424 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:38:32.460 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:38:32.502 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:38:32.599 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:38:32.709 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:38:32.762 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:38:33.522 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:38:33.583 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:38:33.625 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:38:33.674 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:38:33.694 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:38:34.500 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:38:34.500 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:38:34.501 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:38:34.863 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:38:34.863 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:34.863 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:34.863 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:38:34.863 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:38:34.863 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:34.864 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:35.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:38:35.684 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:35.844 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:35.844 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:35.845 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-29T11:38:35.846 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2024-03-29T11:38:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:35.846 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 6.8 M 2024-03-29T11:38:35.846 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 33 M 2024-03-29T11:38:35.846 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:36.517 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-29T11:38:36.611 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T11:38:36.661 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 403 kB/s | 138 kB 00:00 2024-03-29T11:38:37.104 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 124 kB/s | 58 kB 00:00 2024-03-29T11:38:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 109 kB/s | 51 kB 00:00 2024-03-29T11:38:37.304 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 699 kB/s | 548 kB 00:00 2024-03-29T11:38:37.309 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:37.309 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.6 MB/s | 6.8 MB 00:01 2024-03-29T11:38:37.323 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:37.335 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:37.335 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:37.430 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:37.430 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:37.748 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:37.929 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:38:37.981 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:38:38.530 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:38:38.619 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:38:38.688 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:38:38.717 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:38:39.531 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:38:39.531 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:38:39.531 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:38:39.532 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:38:39.532 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:38:39.532 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:38:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:38:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:39.837 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:38:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:38:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:38:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:38:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:38:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:38:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:39.838 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:40.080 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-fuse 2024-03-29T11:38:40.653 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:40.808 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:40.809 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:40.809 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:38:40.809 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:40.809 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:40.809 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 964 k 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 2.8 M 2024-03-29T11:38:40.810 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:40.972 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 486 kB/s | 79 kB 00:00 2024-03-29T11:38:41.114 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 885 kB 00:00 2024-03-29T11:38:41.115 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:41.116 INFO:teuthology.orchestra.run.smithi055.stdout:Total 3.1 MB/s | 964 kB 00:00 2024-03-29T11:38:41.116 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:41.137 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:41.137 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:41.214 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:41.214 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:41.402 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:41.536 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:38:41.617 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:38:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:38:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:38:42.566 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:38:42.566 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:42.566 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:42.566 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:38:42.566 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:42.566 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:42.817 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-volume 2024-03-29T11:38:43.397 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:43.557 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:43.557 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:43.558 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2024-03-29T11:38:43.559 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:43.559 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 332 k 2024-03-29T11:38:43.559 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 1.5 M 2024-03-29T11:38:43.559 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:43.677 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 582 kB/s | 69 kB 00:00 2024-03-29T11:38:43.919 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 732 kB/s | 264 kB 00:00 2024-03-29T11:38:43.920 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:43.921 INFO:teuthology.orchestra.run.smithi055.stdout:Total 916 kB/s | 332 kB 00:00 2024-03-29T11:38:43.921 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:43.930 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:43.930 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:43.968 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:43.969 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:44.120 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:44.317 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:38:44.347 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:38:44.363 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:38:44.363 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:38:44.363 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:38:44.363 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:45.067 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:38:45.385 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:38:45.386 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:45.386 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:45.386 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:38:45.386 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:38:45.386 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:45.386 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:45.712 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados-devel 2024-03-29T11:38:46.293 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:46.450 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 127 k 2024-03-29T11:38:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 456 k 2024-03-29T11:38:46.452 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:46.658 INFO:teuthology.orchestra.run.smithi055.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 612 kB/s | 127 kB 00:00 2024-03-29T11:38:46.659 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:46.659 INFO:teuthology.orchestra.run.smithi055.stdout:Total 608 kB/s | 127 kB 00:00 2024-03-29T11:38:46.659 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:46.665 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:46.665 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:46.722 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:46.723 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:46.874 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:46.967 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:47.395 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:47.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:47.685 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:47.685 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:47.685 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:38:47.685 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:47.685 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:47.910 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs2 2024-03-29T11:38:48.491 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:48.593 INFO:teuthology.orchestra.run.smithi055.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:38:48.650 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:48.651 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-29T11:38:48.651 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:48.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs-devel 2024-03-29T11:38:49.309 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:49.467 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:49.468 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:49.468 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 31 k 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 140 k 2024-03-29T11:38:49.469 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:49.712 INFO:teuthology.orchestra.run.smithi055.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 128 kB/s | 31 kB 00:00 2024-03-29T11:38:49.713 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:49.713 INFO:teuthology.orchestra.run.smithi055.stdout:Total 128 kB/s | 31 kB 00:00 2024-03-29T11:38:49.713 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:49.716 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:49.717 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:49.759 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:49.759 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:49.892 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:49.990 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:50.377 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:50.636 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:50.636 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:50.636 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:50.636 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:38:50.636 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:50.636 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:50.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados2 2024-03-29T11:38:51.377 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:51.478 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:38:51.532 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:51.533 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-29T11:38:51.533 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:51.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librbd1 2024-03-29T11:38:52.157 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:52.257 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:38:52.312 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:52.313 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-29T11:38:52.313 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:52.363 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rados 2024-03-29T11:38:52.941 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:53.042 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:38:53.097 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:53.098 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-29T11:38:53.098 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:53.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rgw 2024-03-29T11:38:53.739 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:53.841 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:38:53.897 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:53.897 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-29T11:38:53.898 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:53.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-cephfs 2024-03-29T11:38:54.529 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:54.631 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:38:54.687 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:54.688 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-29T11:38:54.688 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:54.736 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rbd 2024-03-29T11:38:55.318 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:55.424 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:38:55.481 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:55.482 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-29T11:38:55.482 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:55.542 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-fuse 2024-03-29T11:38:56.118 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:56.273 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:56.273 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 85 k 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 231 k 2024-03-29T11:38:56.274 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:56.599 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 263 kB/s | 85 kB 00:00 2024-03-29T11:38:56.600 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:56.600 INFO:teuthology.orchestra.run.smithi055.stdout:Total 262 kB/s | 85 kB 00:00 2024-03-29T11:38:56.600 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:56.608 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:56.608 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:56.665 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:56.666 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:56.806 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:56.909 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:57.307 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:57.592 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:57.592 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:57.592 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:38:57.592 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:38:57.592 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:57.592 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:38:57.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-mirror 2024-03-29T11:38:58.430 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:38:58.588 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:38:58.589 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:58.589 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-29T11:38:58.589 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:58.589 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:38:58.589 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:38:58.589 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:58.589 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:38:58.589 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:38:58.590 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-29T11:38:58.590 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:38:58.590 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 3.1 M 2024-03-29T11:38:58.590 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 13 M 2024-03-29T11:38:58.590 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:38:59.202 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-03-29T11:38:59.202 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:38:59.203 INFO:teuthology.orchestra.run.smithi055.stdout:Total 5.0 MB/s | 3.1 MB 00:00 2024-03-29T11:38:59.203 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:38:59.212 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:38:59.213 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:38:59.288 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:38:59.288 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:38:59.736 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:38:59.786 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:59.810 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:38:59.811 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:38:59.811 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:38:59.811 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:38:59.811 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:38:59.811 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:39:00.927 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:39:00.927 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:39:00.927 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:39:00.927 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:00.927 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:39:00.927 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:39:01.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-nbd 2024-03-29T11:39:01.734 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:37:10 AM UTC. 2024-03-29T11:39:01.890 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 171 k 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 494 k 2024-03-29T11:39:01.891 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-29T11:39:02.331 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 390 kB/s | 171 kB 00:00 2024-03-29T11:39:02.332 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:39:02.332 INFO:teuthology.orchestra.run.smithi055.stdout:Total 388 kB/s | 171 kB 00:00 2024-03-29T11:39:02.332 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-29T11:39:02.340 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-29T11:39:02.340 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-29T11:39:02.399 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-29T11:39:02.400 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-29T11:39:02.555 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-29T11:39:02.658 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:39:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:39:03.385 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:39:03.385 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:39:03.385 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-29T11:39:03.385 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:03.385 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:39:03.385 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-29T11:39:03.602 DEBUG:teuthology.parallel:result is None 2024-03-29T11:39:35.994 INFO:teuthology.orchestra.run.smithi171.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-29T11:39:35.994 INFO:teuthology.orchestra.run.smithi171.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-29T11:39:36.178 INFO:teuthology.orchestra.run.smithi171.stdout:(31/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 205 B/s | 26 kB 02:09 2024-03-29T11:39:37.569 INFO:teuthology.orchestra.run.smithi171.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 35 kB/s | 4.4 MB 02:10 2024-03-29T11:39:37.572 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:39:37.572 INFO:teuthology.orchestra.run.smithi171.stdout:Total 443 kB/s | 58 MB 02:13 2024-03-29T11:39:37.572 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:39:37.612 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:39:37.612 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:39:38.087 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:39:38.087 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:39:39.593 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:39:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:39:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:39:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:39:39.772 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:39:39.788 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:39:39.981 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:39:40.036 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:39:40.361 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:39:40.434 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:39:40.525 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:39:40.613 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:39:40.644 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:39:40.706 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:39:40.738 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:39:40.779 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:39:40.846 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:39:40.871 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:39:40.943 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:39:40.978 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:39:41.090 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:39:41.121 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:39:41.161 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:39:41.193 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:39:41.222 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:39:41.247 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:39:41.567 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:39:41.593 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:39:41.683 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:39:41.715 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:39:41.758 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:39:42.213 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:39:42.237 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:39:42.299 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:39:42.370 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:39:42.397 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:39:44.514 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:39:44.544 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:39:45.250 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:39:45.289 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:39:45.380 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:39:45.380 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:39:45.380 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:39:45.381 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:39:45.424 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:39:54.795 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:39:54.795 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /sys 2024-03-29T11:39:54.796 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /proc 2024-03-29T11:39:54.796 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /mnt 2024-03-29T11:39:54.796 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /var/tmp 2024-03-29T11:39:54.796 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /home 2024-03-29T11:39:54.796 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /root 2024-03-29T11:39:54.796 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /tmp 2024-03-29T11:39:54.796 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:39:55.707 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:39:55.732 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:39:55.732 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:39:55.732 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:39:55.732 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:39:55.732 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:39:55.732 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:39:55.734 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:39:55.818 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:39:55.818 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:39:56.914 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:39:56.914 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:39:56.914 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:39:56.914 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:39:56.914 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:39:56.914 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:39:56.914 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:39:56.915 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:39:56.916 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:39:56.917 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:39:56.917 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:39:56.917 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:39:56.917 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:39:57.859 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:39:57.859 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:39:57.859 INFO:teuthology.orchestra.run.smithi171.stdout:Upgraded: 2024-03-29T11:39:57.859 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.859 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.860 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:39:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:39:57.862 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:39:57.862 INFO:teuthology.orchestra.run.smithi171.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:39:57.862 INFO:teuthology.orchestra.run.smithi171.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:39:57.862 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:39:57.862 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:39:58.134 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-test 2024-03-29T11:39:58.692 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:39:58.845 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:39:58.845 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:39:58.845 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:39:58.846 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:39:58.847 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2024-03-29T11:39:58.847 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:39:58.847 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 48 M 2024-03-29T11:39:58.847 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 204 M 2024-03-29T11:39:58.847 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:39:59.313 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T11:39:59.355 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-29T11:39:59.388 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:39:59.455 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 478 kB/s | 163 kB 00:00 2024-03-29T11:39:59.547 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:40:01.731 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-29T11:40:01.734 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:40:01.734 INFO:teuthology.orchestra.run.smithi171.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-29T11:40:01.750 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:40:01.771 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:40:01.771 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:40:02.141 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:40:02.141 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:40:02.714 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:40:02.790 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:40:02.827 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:40:02.867 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:40:02.906 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:40:02.933 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:40:07.416 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:40:07.502 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:40:10.013 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:40:10.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:40:10.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:40:10.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:40:10.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:40:10.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:10.381 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:40:10.697 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph 2024-03-29T11:40:11.282 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:50 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:40:11.448 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout:========================================================================================= 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout:========================================================================================= 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:40:11.450 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:40:11.451 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:40:11.452 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:40:11.453 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:40:11.453 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:40:11.453 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:40:11.453 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:11.453 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:40:11.453 INFO:teuthology.orchestra.run.smithi171.stdout:========================================================================================= 2024-03-29T11:40:11.453 INFO:teuthology.orchestra.run.smithi171.stdout:Install 36 Packages 2024-03-29T11:40:11.453 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:11.453 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 30 M 2024-03-29T11:40:11.454 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 107 M 2024-03-29T11:40:11.454 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:40:11.933 INFO:teuthology.orchestra.run.smithi171.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-29T11:40:12.166 INFO:teuthology.orchestra.run.smithi171.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.7 MB/s | 1.5 MB 00:00 2024-03-29T11:40:12.508 INFO:teuthology.orchestra.run.smithi171.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 8.1 MB/s | 4.7 MB 00:00 2024-03-29T11:40:12.550 INFO:teuthology.orchestra.run.smithi171.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 2.1 MB 00:00 2024-03-29T11:40:12.576 INFO:teuthology.orchestra.run.smithi171.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.6 MB/s | 247 kB 00:00 2024-03-29T11:40:12.701 INFO:teuthology.orchestra.run.smithi171.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:40:12.726 INFO:teuthology.orchestra.run.smithi171.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T11:40:12.752 INFO:teuthology.orchestra.run.smithi171.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 627 kB/s | 32 kB 00:00 2024-03-29T11:40:12.777 INFO:teuthology.orchestra.run.smithi171.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-29T11:40:12.803 INFO:teuthology.orchestra.run.smithi171.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2024-03-29T11:40:12.828 INFO:teuthology.orchestra.run.smithi171.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.6 MB/s | 241 kB 00:00 2024-03-29T11:40:12.887 INFO:teuthology.orchestra.run.smithi171.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T11:40:12.912 INFO:teuthology.orchestra.run.smithi171.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T11:40:12.938 INFO:teuthology.orchestra.run.smithi171.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:40:12.963 INFO:teuthology.orchestra.run.smithi171.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:40:13.030 INFO:teuthology.orchestra.run.smithi171.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 334 kB/s | 22 kB 00:00 2024-03-29T11:40:13.089 INFO:teuthology.orchestra.run.smithi171.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 830 kB/s | 125 kB 00:00 2024-03-29T11:40:13.181 INFO:teuthology.orchestra.run.smithi171.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-29T11:40:13.206 INFO:teuthology.orchestra.run.smithi171.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 371 kB/s | 43 kB 00:00 2024-03-29T11:40:13.373 INFO:teuthology.orchestra.run.smithi171.stdout:(20/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 14 MB/s | 17 MB 00:01 2024-03-29T11:40:13.398 INFO:teuthology.orchestra.run.smithi171.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 789 kB/s | 172 kB 00:00 2024-03-29T11:40:13.441 INFO:teuthology.orchestra.run.smithi171.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-29T11:40:13.466 INFO:teuthology.orchestra.run.smithi171.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-03-29T11:40:13.491 INFO:teuthology.orchestra.run.smithi171.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-03-29T11:40:13.517 INFO:teuthology.orchestra.run.smithi171.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 305 kB/s | 23 kB 00:00 2024-03-29T11:40:13.542 INFO:teuthology.orchestra.run.smithi171.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-29T11:40:13.567 INFO:teuthology.orchestra.run.smithi171.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-29T11:40:13.593 INFO:teuthology.orchestra.run.smithi171.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 611 kB/s | 46 kB 00:00 2024-03-29T11:40:13.618 INFO:teuthology.orchestra.run.smithi171.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T11:40:13.643 INFO:teuthology.orchestra.run.smithi171.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-03-29T11:40:13.677 INFO:teuthology.orchestra.run.smithi171.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-29T11:40:13.702 INFO:teuthology.orchestra.run.smithi171.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 607 kB/s | 36 kB 00:00 2024-03-29T11:40:13.728 INFO:teuthology.orchestra.run.smithi171.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-03-29T11:40:13.753 INFO:teuthology.orchestra.run.smithi171.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T11:40:13.778 INFO:teuthology.orchestra.run.smithi171.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-03-29T11:40:13.812 INFO:teuthology.orchestra.run.smithi171.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-29T11:40:13.816 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:40:13.816 INFO:teuthology.orchestra.run.smithi171.stdout:Total 13 MB/s | 30 MB 00:02 2024-03-29T11:40:13.841 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:40:13.883 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:40:13.883 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:40:14.298 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:40:14.298 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:40:15.636 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:40:15.690 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:40:15.732 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:40:15.760 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:40:15.830 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:40:15.875 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:40:15.908 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:40:16.002 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:40:16.070 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:40:16.112 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:40:16.184 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:40:16.239 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:40:16.276 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:40:16.326 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:40:16.382 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:40:16.577 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:40:16.632 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:40:16.671 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:40:16.706 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:40:16.736 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:40:16.813 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:40:16.847 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:40:16.880 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:40:16.920 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:40:16.956 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:40:17.007 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:40:17.105 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:40:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:40:17.219 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:40:17.358 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:40:17.485 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:40:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:40:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:40:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:40:17.714 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:40:17.714 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:40:17.714 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:40:17.714 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:40:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:19.247 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:40:19.274 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:40:19.274 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:40:19.274 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:40:19.274 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:40:19.275 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:40:19.275 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:19.715 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:40:19.738 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:40:19.739 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:40:19.739 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:40:19.739 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:40:19.739 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:40:19.739 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:19.949 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:40:19.972 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:40:19.972 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:40:19.972 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:40:19.972 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:40:19.972 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:40:19.972 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:20.039 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:40:21.651 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:40:21.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:40:21.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:40:21.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:40:21.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:40:21.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:40:21.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:40:21.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:40:21.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:40:21.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:40:21.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:40:21.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:40:21.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:40:21.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:40:21.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:40:21.655 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:40:22.550 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:40:22.550 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:22.550 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:40:22.550 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:40:22.550 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:40:22.550 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:40:22.550 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:40:22.550 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:40:22.550 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:40:22.551 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:40:22.552 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:40:22.553 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:22.553 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:40:22.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-base 2024-03-29T11:40:23.476 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:02 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:40:23.575 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:40:23.628 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:40:23.629 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-29T11:40:23.629 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:40:23.704 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install cephadm 2024-03-29T11:40:24.271 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:03 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:40:24.422 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 747 k 2024-03-29T11:40:24.423 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 754 k 2024-03-29T11:40:24.424 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:40:24.725 INFO:teuthology.orchestra.run.smithi171.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.4 MB/s | 747 kB 00:00 2024-03-29T11:40:24.725 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:40:24.725 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.4 MB/s | 747 kB 00:00 2024-03-29T11:40:24.726 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:40:24.730 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:40:24.731 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:40:24.736 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:40:24.737 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:40:25.225 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:40:25.298 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:40:25.685 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:40:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:40:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:25.954 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:40:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:40:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:25.954 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:40:26.173 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:40:26.741 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:05 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:40:26.896 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:40:26.897 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:40:26.897 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-29T11:40:26.897 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:40:26.897 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:40:26.898 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:40:26.898 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:26.898 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:40:26.898 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:40:26.898 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-29T11:40:26.898 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:26.898 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 145 k 2024-03-29T11:40:26.898 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 436 k 2024-03-29T11:40:26.898 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:40:27.243 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 420 kB/s | 145 kB 00:00 2024-03-29T11:40:27.244 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:40:27.244 INFO:teuthology.orchestra.run.smithi171.stdout:Total 419 kB/s | 145 kB 00:00 2024-03-29T11:40:27.244 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:40:27.252 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:40:27.252 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:40:27.304 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:40:27.305 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:40:27.448 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:40:27.503 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:40:27.528 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:40:27.528 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:40:27.528 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:40:27.529 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:28.511 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:40:28.511 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:28.511 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:40:28.512 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:40:28.512 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:28.512 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:40:28.786 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr 2024-03-29T11:40:29.363 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:08 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:40:29.461 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:40:29.514 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:40:29.515 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-29T11:40:29.516 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:40:29.594 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:40:30.154 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:09 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:40:30.303 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:40:30.305 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2024-03-29T11:40:30.305 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:40:30.305 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2024-03-29T11:40:30.305 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:40:30.305 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:40:30.305 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:40:30.306 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:40:30.306 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:40:30.306 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:40:30.306 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:40:30.306 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:40:30.306 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:40:30.307 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:40:30.307 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:40:30.307 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:40:30.307 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:40:30.307 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:40:30.307 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:30.308 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:40:30.308 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2024-03-29T11:40:30.308 INFO:teuthology.orchestra.run.smithi171.stdout:Install 12 Packages 2024-03-29T11:40:30.308 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:30.309 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 6.9 M 2024-03-29T11:40:30.309 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 26 M 2024-03-29T11:40:30.309 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:40:30.830 INFO:teuthology.orchestra.run.smithi171.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 78 kB/s | 15 kB 00:00 2024-03-29T11:40:30.886 INFO:teuthology.orchestra.run.smithi171.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 123 kB/s | 31 kB 00:00 2024-03-29T11:40:30.986 INFO:teuthology.orchestra.run.smithi171.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.3 MB/s | 1.0 MB 00:00 2024-03-29T11:40:31.011 INFO:teuthology.orchestra.run.smithi171.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-29T11:40:31.162 INFO:teuthology.orchestra.run.smithi171.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-03-29T11:40:31.204 INFO:teuthology.orchestra.run.smithi171.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.1 MB/s | 1.8 MB 00:00 2024-03-29T11:40:31.388 INFO:teuthology.orchestra.run.smithi171.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 87 kB/s | 19 kB 00:00 2024-03-29T11:40:31.496 INFO:teuthology.orchestra.run.smithi171.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:40:31.588 INFO:teuthology.orchestra.run.smithi171.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 5.3 MB/s | 2.0 MB 00:00 2024-03-29T11:40:31.605 INFO:teuthology.orchestra.run.smithi171.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 665 kB/s | 144 kB 00:00 2024-03-29T11:40:31.622 INFO:teuthology.orchestra.run.smithi171.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 246 kB/s | 31 kB 00:00 2024-03-29T11:40:31.655 INFO:teuthology.orchestra.run.smithi171.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T11:40:31.659 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:40:31.659 INFO:teuthology.orchestra.run.smithi171.stdout:Total 5.1 MB/s | 6.9 MB 00:01 2024-03-29T11:40:31.677 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:40:31.694 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:40:31.694 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:40:31.867 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:40:31.868 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:40:32.382 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:40:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:40:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:40:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:40:32.578 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:40:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:40:32.815 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:40:32.917 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:40:32.959 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:40:32.997 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:40:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:40:33.272 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:40:33.304 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:40:34.100 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:40:34.100 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:40:34.100 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:40:34.100 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:40:34.100 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:40:34.100 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:40:34.100 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:40:34.100 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:40:34.101 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:40:34.101 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:40:34.101 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:40:34.101 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:40:34.455 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:34.456 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:40:34.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:40:35.273 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:14 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:40:35.426 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:40:35.427 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-03-29T11:40:35.427 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:40:35.427 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-03-29T11:40:35.427 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:40:35.427 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:40:35.427 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:40:35.427 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:40:35.427 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:40:35.427 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout:Install 12 Packages 2024-03-29T11:40:35.428 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:35.429 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 37 M 2024-03-29T11:40:35.429 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 209 M 2024-03-29T11:40:35.429 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:40:35.717 INFO:teuthology.orchestra.run.smithi171.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 234 kB/s | 32 kB 00:00 2024-03-29T11:40:35.743 INFO:teuthology.orchestra.run.smithi171.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 712 kB/s | 17 kB 00:00 2024-03-29T11:40:35.768 INFO:teuthology.orchestra.run.smithi171.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-03-29T11:40:35.935 INFO:teuthology.orchestra.run.smithi171.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T11:40:35.994 INFO:teuthology.orchestra.run.smithi171.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.3 MB/s | 3.0 MB 00:00 2024-03-29T11:40:36.078 INFO:teuthology.orchestra.run.smithi171.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-29T11:40:36.103 INFO:teuthology.orchestra.run.smithi171.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-03-29T11:40:36.429 INFO:teuthology.orchestra.run.smithi171.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-29T11:40:36.612 INFO:teuthology.orchestra.run.smithi171.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 7.2 MB/s | 7.4 MB 00:01 2024-03-29T11:40:36.637 INFO:teuthology.orchestra.run.smithi171.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 937 kB/s | 195 kB 00:00 2024-03-29T11:40:36.671 INFO:teuthology.orchestra.run.smithi171.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-29T11:40:37.221 INFO:teuthology.orchestra.run.smithi171.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 403 kB/s | 245 kB 00:00 2024-03-29T11:40:37.223 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:40:37.224 INFO:teuthology.orchestra.run.smithi171.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T11:40:37.268 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:40:37.293 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:40:37.294 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:40:37.659 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:40:37.660 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:40:38.361 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:40:38.445 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:40:38.484 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:40:38.578 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:40:38.621 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:40:38.802 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:40:38.829 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:40:38.906 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:40:39.560 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:40:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:40:40.904 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:40:41.960 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:40:44.501 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:40:44.948 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:40:44.948 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:44.948 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:40:44.948 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:40:44.948 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:44.949 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:40:45.377 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:40:45.951 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:24 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:40:46.103 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:40:46.104 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-29T11:40:46.104 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:40:46.104 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-29T11:40:46.104 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:40:46.104 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:40:46.104 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:40:46.104 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:40:46.104 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:40:46.104 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-29T11:40:46.105 INFO:teuthology.orchestra.run.smithi171.stdout:Install 13 Packages 2024-03-29T11:40:46.106 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:46.106 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 2.0 M 2024-03-29T11:40:46.106 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 26 M 2024-03-29T11:40:46.106 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:40:46.308 INFO:teuthology.orchestra.run.smithi171.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 24 kB 00:00 2024-03-29T11:40:46.333 INFO:teuthology.orchestra.run.smithi171.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2024-03-29T11:40:46.375 INFO:teuthology.orchestra.run.smithi171.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:40:46.401 INFO:teuthology.orchestra.run.smithi171.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:40:46.426 INFO:teuthology.orchestra.run.smithi171.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:40:46.451 INFO:teuthology.orchestra.run.smithi171.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:40:46.476 INFO:teuthology.orchestra.run.smithi171.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 180 kB/s | 52 kB 00:00 2024-03-29T11:40:46.577 INFO:teuthology.orchestra.run.smithi171.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 119 kB/s | 15 kB 00:00 2024-03-29T11:40:46.602 INFO:teuthology.orchestra.run.smithi171.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2024-03-29T11:40:46.694 INFO:teuthology.orchestra.run.smithi171.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 932 kB/s | 203 kB 00:00 2024-03-29T11:40:46.720 INFO:teuthology.orchestra.run.smithi171.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-03-29T11:40:46.762 INFO:teuthology.orchestra.run.smithi171.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-29T11:40:46.845 INFO:teuthology.orchestra.run.smithi171.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-29T11:40:46.847 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:40:46.847 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-29T11:40:46.847 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:40:46.861 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:40:46.861 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:40:46.953 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:40:46.953 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:40:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:40:47.633 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:40:47.691 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:40:47.755 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:40:47.792 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:40:47.833 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:40:47.927 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:40:48.037 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:40:48.086 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:40:48.812 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:40:48.859 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:40:48.893 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:40:48.942 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:40:48.963 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:40:49.764 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:40:49.765 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:40:50.134 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:40:50.134 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:50.134 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:40:50.135 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:40:50.135 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:40:50.135 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:40:50.135 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:40:50.135 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:40:50.135 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:40:50.135 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:40:50.135 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:40:50.135 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:40:50.136 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:40:50.136 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:40:50.136 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:40:50.136 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:40:50.136 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:50.136 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:40:50.384 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:40:50.956 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:29 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:40:51.107 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:40:51.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:40:51.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:40:51.109 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:51.109 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:40:51.109 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-29T11:40:51.109 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2024-03-29T11:40:51.109 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:40:51.109 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 6.8 M 2024-03-29T11:40:51.109 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 33 M 2024-03-29T11:40:51.110 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:40:51.680 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T11:40:51.705 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 486 kB/s | 138 kB 00:00 2024-03-29T11:40:52.089 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 371 kB/s | 248 kB 00:00 2024-03-29T11:40:52.247 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-03-29T11:40:54.939 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 18 kB/s | 58 kB 00:03 2024-03-29T11:40:59.063 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 74 kB/s | 548 kB 00:07 2024-03-29T11:40:59.065 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:40:59.065 INFO:teuthology.orchestra.run.smithi171.stdout:Total 875 kB/s | 6.8 MB 00:07 2024-03-29T11:40:59.081 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:40:59.092 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:40:59.093 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:40:59.185 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:40:59.186 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:40:59.488 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:40:59.664 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:40:59.716 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:41:00.229 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:41:00.310 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:41:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:41:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:41:01.194 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:41:01.194 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:41:01.195 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:41:01.195 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:41:01.195 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:41:01.195 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:41:01.509 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:41:01.509 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:01.510 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:41:01.510 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:41:01.510 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:41:01.510 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:41:01.510 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:41:01.510 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:41:01.510 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:41:01.510 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:01.510 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:01.760 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-fuse 2024-03-29T11:41:02.324 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:41 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:02.476 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:02.476 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:02.476 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:41:02.476 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:02.476 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:41:02.476 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:41:02.476 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:41:02.476 INFO:teuthology.orchestra.run.smithi171.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:41:02.477 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:02.477 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:41:02.477 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:02.477 INFO:teuthology.orchestra.run.smithi171.stdout:Install 2 Packages 2024-03-29T11:41:02.477 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:02.477 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 964 k 2024-03-29T11:41:02.477 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 2.8 M 2024-03-29T11:41:02.478 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:41:02.643 INFO:teuthology.orchestra.run.smithi171.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 475 kB/s | 79 kB 00:00 2024-03-29T11:41:02.818 INFO:teuthology.orchestra.run.smithi171.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.5 MB/s | 885 kB 00:00 2024-03-29T11:41:02.819 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:41:02.820 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.7 MB/s | 964 kB 00:00 2024-03-29T11:41:02.820 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:41:02.840 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:41:02.840 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:41:02.914 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:41:02.914 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:41:03.110 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:41:03.244 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:41:03.309 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:41:04.029 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:41:04.029 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:41:04.331 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:41:04.331 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:04.332 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:41:04.332 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:41:04.332 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:04.332 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:04.592 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-volume 2024-03-29T11:41:05.177 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:44 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:05.330 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:05.330 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:05.330 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-29T11:41:05.330 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:05.330 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:41:05.330 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:41:05.330 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-29T11:41:05.331 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:41:05.331 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:05.331 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:41:05.331 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:05.331 INFO:teuthology.orchestra.run.smithi171.stdout:Install 2 Packages 2024-03-29T11:41:05.331 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:05.332 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 332 k 2024-03-29T11:41:05.332 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 1.5 M 2024-03-29T11:41:05.332 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:41:05.461 INFO:teuthology.orchestra.run.smithi171.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 534 kB/s | 69 kB 00:00 2024-03-29T11:41:05.736 INFO:teuthology.orchestra.run.smithi171.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 653 kB/s | 264 kB 00:00 2024-03-29T11:41:05.737 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:41:05.737 INFO:teuthology.orchestra.run.smithi171.stdout:Total 818 kB/s | 332 kB 00:00 2024-03-29T11:41:05.737 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:41:05.746 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:41:05.746 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:41:05.784 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:41:05.784 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:41:05.933 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:41:06.110 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:41:06.156 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:41:06.172 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:41:06.172 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:41:06.172 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:41:06.172 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:06.904 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:41:07.248 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:41:07.248 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:07.248 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:41:07.248 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:41:07.248 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:41:07.248 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:07.248 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:07.611 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados-devel 2024-03-29T11:41:08.196 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:47 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:08.346 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:08.347 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:08.347 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-29T11:41:08.347 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:08.347 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:41:08.347 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:41:08.347 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:08.347 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:41:08.348 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:08.348 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-29T11:41:08.348 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:08.348 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 127 k 2024-03-29T11:41:08.348 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 456 k 2024-03-29T11:41:08.348 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:41:08.721 INFO:teuthology.orchestra.run.smithi171.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 339 kB/s | 127 kB 00:00 2024-03-29T11:41:08.722 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:41:08.722 INFO:teuthology.orchestra.run.smithi171.stdout:Total 338 kB/s | 127 kB 00:00 2024-03-29T11:41:08.722 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:41:08.729 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:41:08.729 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:41:08.784 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:41:08.784 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:41:08.925 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:41:09.020 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:09.797 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:09.798 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:09.798 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:41:09.798 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:41:09.798 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:09.798 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:10.018 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs2 2024-03-29T11:41:10.601 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:49 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:10.706 INFO:teuthology.orchestra.run.smithi171.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:41:10.760 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:10.761 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-29T11:41:10.761 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:10.835 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs-devel 2024-03-29T11:41:11.403 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:50 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:11.554 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:11.555 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:11.555 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-29T11:41:11.555 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:11.555 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:41:11.555 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:41:11.555 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:11.556 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:41:11.556 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:11.556 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-29T11:41:11.556 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:11.556 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 31 k 2024-03-29T11:41:11.556 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 140 k 2024-03-29T11:41:11.556 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:41:11.728 INFO:teuthology.orchestra.run.smithi171.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 181 kB/s | 31 kB 00:00 2024-03-29T11:41:11.728 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:41:11.729 INFO:teuthology.orchestra.run.smithi171.stdout:Total 180 kB/s | 31 kB 00:00 2024-03-29T11:41:11.729 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:41:11.732 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:41:11.732 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:41:11.760 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:41:11.760 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:41:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:41:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:12.341 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:12.622 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:12.622 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:12.622 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:41:12.622 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:41:12.622 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:12.622 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:12.784 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados2 2024-03-29T11:41:13.369 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:52 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:13.471 INFO:teuthology.orchestra.run.smithi171.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:41:13.527 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:13.528 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-29T11:41:13.528 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:13.581 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librbd1 2024-03-29T11:41:14.171 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:53 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:14.273 INFO:teuthology.orchestra.run.smithi171.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:41:14.326 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:14.328 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-29T11:41:14.328 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:14.381 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rados 2024-03-29T11:41:14.949 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:53 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:15.047 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:41:15.099 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:15.100 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-29T11:41:15.100 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:15.162 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rgw 2024-03-29T11:41:15.730 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:54 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:15.828 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:41:15.880 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:15.881 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-29T11:41:15.881 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:15.933 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-cephfs 2024-03-29T11:41:16.505 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:55 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:16.604 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:41:16.657 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:16.658 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-29T11:41:16.658 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:16.711 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rbd 2024-03-29T11:41:17.298 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:56 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:17.396 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:41:17.448 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:17.449 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-29T11:41:17.449 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:17.513 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-fuse 2024-03-29T11:41:18.078 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:57 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:18.226 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:18.226 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:18.226 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 85 k 2024-03-29T11:41:18.227 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 231 k 2024-03-29T11:41:18.228 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:41:18.441 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 400 kB/s | 85 kB 00:00 2024-03-29T11:41:18.442 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:41:18.442 INFO:teuthology.orchestra.run.smithi171.stdout:Total 397 kB/s | 85 kB 00:00 2024-03-29T11:41:18.442 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:41:18.450 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:41:18.450 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:41:18.504 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:41:18.505 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:41:18.647 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:41:18.740 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:19.136 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:19.438 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:19.438 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:19.438 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:41:19.438 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:41:19.439 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:19.439 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:19.687 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-mirror 2024-03-29T11:41:20.258 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:59 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:20.413 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:20.414 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:20.414 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-29T11:41:20.414 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:20.414 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:41:20.415 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:41:20.415 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:20.415 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:41:20.415 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:20.415 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-29T11:41:20.415 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:20.415 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 3.1 M 2024-03-29T11:41:20.415 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 13 M 2024-03-29T11:41:20.415 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:41:20.940 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.9 MB/s | 3.1 MB 00:00 2024-03-29T11:41:20.940 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:41:20.941 INFO:teuthology.orchestra.run.smithi171.stdout:Total 5.9 MB/s | 3.1 MB 00:00 2024-03-29T11:41:20.941 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:41:20.950 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:41:20.950 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:41:21.022 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:41:21.022 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:41:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:41:21.524 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:21.549 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:21.549 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:41:21.549 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:41:21.549 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:41:21.549 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:41:21.549 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:22.679 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:22.679 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:22.679 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:41:22.679 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:41:22.679 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:22.679 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:22.966 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-nbd 2024-03-29T11:41:23.546 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:04:02 ago on Fri 29 Mar 2024 11:37:21 AM UTC. 2024-03-29T11:41:23.694 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-29T11:41:23.695 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:23.695 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-03-29T11:41:23.695 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 171 k 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 494 k 2024-03-29T11:41:23.696 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-29T11:41:24.005 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 556 kB/s | 171 kB 00:00 2024-03-29T11:41:24.005 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:41:24.006 INFO:teuthology.orchestra.run.smithi171.stdout:Total 553 kB/s | 171 kB 00:00 2024-03-29T11:41:24.006 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-29T11:41:24.014 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-29T11:41:24.014 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-29T11:41:24.070 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-29T11:41:24.070 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-29T11:41:24.225 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-29T11:41:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:24.731 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:25.044 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:41:25.044 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:25.044 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-29T11:41:25.044 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:41:25.044 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:25.044 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-29T11:41:25.269 DEBUG:teuthology.parallel:result is None 2024-03-29T11:41:25.270 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:41:25.419 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:41:25.445 INFO:teuthology.orchestra.run.smithi055.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:41:25.446 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:41:25.446 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:41:25.448 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:41:25.665 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:41:25.686 INFO:teuthology.orchestra.run.smithi171.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:41:25.686 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:41:25.686 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:41:25.688 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:41:25.688 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:25.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:41:25.721 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:41:25.722 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:41:25.754 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:41:25.755 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:25.755 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:41:25.792 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:41:25.862 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:41:25.862 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:41:25.890 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:41:25.958 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:41:25.958 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:25.958 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:41:25.990 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:41:26.061 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:41:26.061 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:41:26.088 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:41:26.156 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:41:26.156 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:26.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:41:26.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:41:26.256 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:41:26.256 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:41:26.283 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:41:26.352 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:41:26.443 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:41:26.444 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /var/log/ceph 2024-03-29T11:41:26.446 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /var/log/ceph 2024-03-29T11:41:26.474 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:41:26.474 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:41:26.517 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:41:26.544 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:41:26.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:41:26.586 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:41:26.625 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:41:26.625 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:41:26.626 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622733', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi055.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDn8hr+R//BuBItQCC5dVoSKBKDvgSrKDS+QH9rlJSAYWFy2nL/xzTn7JCmO18nSwnYdEASNcybaXfNAOMWEN/4=', 'smithi171.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOFSxlvW3aKHYhb1nDKtEDx052k4LRqJy3lWvFvDkLSuD/IqfruqCmgITtVgCg5NNInCZA9x6Fe/yzLIJ4Iktqs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:41:26.626 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:41:26.656 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:41:26.670 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:41:26.712 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:41:26.739 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:26.739 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:41:26.793 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:41:26.793 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2024-03-29T11:41:26.848 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:41:26.848 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:41:26.848 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:41:26.848 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:41:26.848 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:41:26.848 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-29 11:39:00.041388289 +0000 2024-03-29T11:41:26.848 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-29 11:36:27.026053474 +0000 2024-03-29T11:41:26.848 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-29 11:36:27.026053474 +0000 2024-03-29T11:41:26.849 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-29 11:36:27.026053474 +0000 2024-03-29T11:41:26.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:41:26.913 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-29T11:41:26.914 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-29T11:41:26.914 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000290135 s, 1.8 MB/s 2024-03-29T11:41:26.915 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:41:26.971 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2024-03-29T11:41:27.025 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:41:27.026 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:41:27.026 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:41:27.026 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:41:27.026 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:41:27.026 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-29 11:39:00.041388289 +0000 2024-03-29T11:41:27.026 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-29 11:36:27.392046884 +0000 2024-03-29T11:41:27.026 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-29 11:36:27.392046884 +0000 2024-03-29T11:41:27.026 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-29 11:36:27.392046884 +0000 2024-03-29T11:41:27.026 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:41:27.092 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-29T11:41:27.092 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-29T11:41:27.092 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000280264 s, 1.8 MB/s 2024-03-29T11:41:27.092 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:41:27.150 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2024-03-29T11:41:27.204 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:41:27.204 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:41:27.205 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:41:27.205 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:41:27.205 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:41:27.205 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-29 11:39:00.042388271 +0000 2024-03-29T11:41:27.205 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-29 11:36:27.750040439 +0000 2024-03-29T11:41:27.205 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-29 11:36:27.750040439 +0000 2024-03-29T11:41:27.205 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-29 11:36:27.750040439 +0000 2024-03-29T11:41:27.205 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:41:27.269 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-29T11:41:27.269 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-29T11:41:27.269 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000245306 s, 2.1 MB/s 2024-03-29T11:41:27.270 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:41:27.326 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2024-03-29T11:41:27.381 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:41:27.381 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:41:27.381 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:41:27.381 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:41:27.381 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:41:27.381 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-29 11:39:00.042388271 +0000 2024-03-29T11:41:27.381 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-29 11:36:28.122033742 +0000 2024-03-29T11:41:27.381 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-29 11:36:28.122033742 +0000 2024-03-29T11:41:27.381 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-29 11:36:28.122033742 +0000 2024-03-29T11:41:27.382 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:41:27.445 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-29T11:41:27.445 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-29T11:41:27.445 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000288405 s, 1.8 MB/s 2024-03-29T11:41:27.446 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:41:27.501 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:27.501 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:41:27.501 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:41:27.518 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:41:27.518 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_1 2024-03-29T11:41:27.574 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:41:27.574 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:41:27.574 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:41:27.574 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:41:27.574 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:41:27.574 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-29 11:41:21.786061722 +0000 2024-03-29T11:41:27.574 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-29 11:36:18.669571807 +0000 2024-03-29T11:41:27.575 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-29 11:36:18.669571807 +0000 2024-03-29T11:41:27.575 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-03-29 11:36:18.669571807 +0000 2024-03-29T11:41:27.575 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:41:27.640 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-29T11:41:27.640 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-29T11:41:27.640 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000205776 s, 2.5 MB/s 2024-03-29T11:41:27.642 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:41:27.698 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_2 2024-03-29T11:41:27.754 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:41:27.754 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:41:27.754 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:41:27.754 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:41:27.754 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:41:27.754 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-29 11:41:21.786061722 +0000 2024-03-29T11:41:27.754 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-29 11:36:19.015564762 +0000 2024-03-29T11:41:27.754 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-29 11:36:19.015564762 +0000 2024-03-29T11:41:27.754 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-03-29 11:36:19.015564762 +0000 2024-03-29T11:41:27.755 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:41:27.820 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-29T11:41:27.820 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-29T11:41:27.820 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000230097 s, 2.2 MB/s 2024-03-29T11:41:27.821 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:41:27.877 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_3 2024-03-29T11:41:27.933 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:41:27.933 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:41:27.933 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:41:27.933 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:41:27.933 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:41:27.934 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-29 11:41:21.786061722 +0000 2024-03-29T11:41:27.934 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-29 11:36:19.354557860 +0000 2024-03-29T11:41:27.934 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-29 11:36:19.354557860 +0000 2024-03-29T11:41:27.934 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-03-29 11:36:19.354557860 +0000 2024-03-29T11:41:27.934 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:41:27.999 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-29T11:41:27.999 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-29T11:41:27.999 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000224908 s, 2.3 MB/s 2024-03-29T11:41:28.000 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:41:28.055 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_4 2024-03-29T11:41:28.110 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:41:28.110 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:41:28.110 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:41:28.110 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:41:28.111 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:41:28.111 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-29 11:41:21.787061701 +0000 2024-03-29T11:41:28.111 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-29 11:36:19.693550958 +0000 2024-03-29T11:41:28.111 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-29 11:36:19.693550958 +0000 2024-03-29T11:41:28.111 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-03-29 11:36:19.693550958 +0000 2024-03-29T11:41:28.111 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:41:28.175 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-29T11:41:28.175 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-29T11:41:28.175 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000225277 s, 2.3 MB/s 2024-03-29T11:41:28.176 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:41:28.232 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:28.232 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi171.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:41:28.232 INFO:tasks.ceph:Generating config... 2024-03-29T11:41:28.233 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:41:28.234 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:41:28.234 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:41:28.234 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:41:28.234 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:41:28.234 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:41:28.234 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:41:28.234 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:41:28.234 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:41:28.234 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:41:28.234 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:41:28.234 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:41:28.234 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:41:28.234 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:41:28.234 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:41:28.235 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:41:28.235 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:41:28.235 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:41:28.235 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:41:28.235 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:41:28.235 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:41:28.235 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:41:28.235 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:41:28.235 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:41:28.235 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:41:28.235 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:41:28.235 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:41:28.235 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:41:28.235 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:41:28.235 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:41:28.236 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:41:28.237 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:41:28.237 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:41:28.237 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:41:28.237 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:41:28.291 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:41:28.293 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:41:28.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:41:28.370 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.55:6789]'), ('mon.c', '[v1:172.21.15.55:6790]'), ('mon.b', '[v1:172.21.15.171:6789]')] 2024-03-29T11:41:28.371 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.55:6789],[v1:172.21.15.55:6790],[v1:172.21.15.171:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:41:28.372 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:28.372 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:41:28.425 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.55:6789]' --addv c '[v1:172.21.15.55:6790]' --addv b '[v1:172.21.15.171:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: generated fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:setting min_mon_release = quincy 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 0 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:min_mon_release 17 (quincy) 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:election_strategy: 1 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:41:28.511 INFO:teuthology.orchestra.run.smithi055.stdout:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:41:28.512 INFO:teuthology.orchestra.run.smithi055.stdout:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:41:28.512 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:41:28.513 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:41:28.567 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4d42039a-57d6-490b-b589-aba66145591c... 2024-03-29T11:41:28.568 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:41:28.611 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:41:28.658 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: chdir = "" 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: auth supported = cephx 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: filestore xattr use omap = true 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: mon clock drift allowed = 1.000 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: osd crush chooseleaf type = 0 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: auth debug = true 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on old message = true 2024-03-29T11:41:28.659 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on bug = true 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: mon pg warn max object skew = 0 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default size = 2 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow primary affinity = true 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow pg remap = true 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:41:28.660 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on no sortbitwise = false 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on too few osds = false 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: mon_allow_pool_size_one = true 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: osd default data pool replay window = 5 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: mon allow pool delete = true 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: mon cluster log file level = debug 2024-03-29T11:41:28.661 INFO:teuthology.orchestra.run.smithi055.stdout: debug asserts on shutdown = true 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: mon health detail to clog = false 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: mon host = "[v1:172.21.15.55:6789],[v1:172.21.15.55:6790],[v1:172.21.15.171:6789]" 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: lockdep = True 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: mon client directed command retry = 5 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: mon election default strategy = 1 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: ms bind msgr2 = False 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: ms inject socket failures = 5000 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: ms type = async 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout:[osd] 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: osd journal size = 100 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.662 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub load threshold = 5.0 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub max interval = 600 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd recover clone overlap = true 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd recovery max chunk = 1048576 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug shutdown = true 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug op order = true 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify stray on activate = true 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug trim objects = true 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd open classes on start = true 2024-03-29T11:41:28.663 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug pg log writeout = true 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: osd map max advance = 10 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: journal zero on create = true 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: filestore apply finisher threads = 3 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: bdev debug aio = true 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug misdirected ops = true 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore block size = 96636764160 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore compression algorithm = snappy 2024-03-29T11:41:28.664 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore compression mode = aggressive 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore fsck on mount = True 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore zero block detection = True 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluefs = 20 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluestore = 20 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: debug monc = 20 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: debug osd = 20 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: debug rocksdb = 10 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd full ratio = 0.9 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: osd class default list = * 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: osd class load list = * 2024-03-29T11:41:28.665 INFO:teuthology.orchestra.run.smithi055.stdout: osd client watch timeout = 120 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify cached snaps = True 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify missing on start = True 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: osd mclock override recovery settings = True 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: osd objectstore = bluestore 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: osd op queue = debug_random 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: osd op queue cut off = debug_random 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: osd shutdown pgref assert = True 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr] 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-29T11:41:28.666 INFO:teuthology.orchestra.run.smithi055.stdout: debug mgr = 20 2024-03-29T11:41:28.667 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2024-03-29T11:41:28.667 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2024-03-29T11:41:28.667 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:41:28.667 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:41:28.667 INFO:teuthology.orchestra.run.smithi055.stdout: mgr/telemetry/nag = false 2024-03-29T11:41:28.667 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.667 INFO:teuthology.orchestra.run.smithi055.stdout:[mon] 2024-03-29T11:41:28.667 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-29T11:41:28.667 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: debug paxos = 20 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: mon data avail warn = 5 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: mon mgr mkfs grace = 240 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd reporter subtree level = osd 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd prime pg temp = true 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:41:28.668 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: # 1m isn't quite enough 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: mon min osdmap epochs = 50 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: mon osdmap full prune interval = 2 2024-03-29T11:41:28.669 INFO:teuthology.orchestra.run.smithi055.stdout: mon osdmap full prune min = 15 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: mon scrub interval = 300 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on pool no app = False 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: paxos service trim min = 10 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout:[client] 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: rgw cache enabled = true 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable ops log = true 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable usage log = true 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: debug monc = 20 2024-03-29T11:41:28.670 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-29T11:41:28.671 INFO:teuthology.orchestra.run.smithi055.stdout: debug objecter = 20 2024-03-29T11:41:28.671 INFO:teuthology.orchestra.run.smithi055.stdout: debug rados = 20 2024-03-29T11:41:28.671 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.a] 2024-03-29T11:41:28.671 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.c] 2024-03-29T11:41:28.671 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.b] 2024-03-29T11:41:28.671 INFO:teuthology.orchestra.run.smithi171.stdout:[global] 2024-03-29T11:41:28.671 INFO:teuthology.orchestra.run.smithi171.stdout: chdir = "" 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: auth supported = cephx 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: filestore xattr use omap = true 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: mon clock drift allowed = 1.000 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: osd crush chooseleaf type = 0 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: auth debug = true 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on old message = true 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on bug = true 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:41:28.672 INFO:teuthology.orchestra.run.smithi171.stdout: mon pg warn max object skew = 0 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default size = 2 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow primary affinity = true 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow pg remap = true 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on no sortbitwise = false 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on too few osds = false 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:41:28.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: mon_allow_pool_size_one = true 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: osd default data pool replay window = 5 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: mon allow pool delete = true 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: mon cluster log file level = debug 2024-03-29T11:41:28.674 INFO:teuthology.orchestra.run.smithi171.stdout: debug asserts on shutdown = true 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: mon health detail to clog = false 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: mon host = "[v1:172.21.15.55:6789],[v1:172.21.15.55:6790],[v1:172.21.15.171:6789]" 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: lockdep = True 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: mon client directed command retry = 5 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: mon election default strategy = 1 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: ms bind msgr2 = False 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: ms inject socket failures = 5000 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: ms type = async 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: fsid = 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout:[osd] 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: osd journal size = 100 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub load threshold = 5.0 2024-03-29T11:41:28.675 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub max interval = 600 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd recover clone overlap = true 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd recovery max chunk = 1048576 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug shutdown = true 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug op order = true 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify stray on activate = true 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug trim objects = true 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd open classes on start = true 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug pg log writeout = true 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.676 INFO:teuthology.orchestra.run.smithi171.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:41:28.677 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.677 INFO:teuthology.orchestra.run.smithi171.stdout: osd map max advance = 10 2024-03-29T11:41:28.677 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.677 INFO:teuthology.orchestra.run.smithi171.stdout: journal zero on create = true 2024-03-29T11:41:28.677 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.677 INFO:teuthology.orchestra.run.smithi171.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: filestore apply finisher threads = 3 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: bdev debug aio = true 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug misdirected ops = true 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore block size = 96636764160 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore compression algorithm = snappy 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore compression mode = aggressive 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore fsck on mount = True 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore zero block detection = True 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluefs = 20 2024-03-29T11:41:28.678 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluestore = 20 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: debug monc = 20 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: debug osd = 20 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: debug rocksdb = 10 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd full ratio = 0.9 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: osd class default list = * 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: osd class load list = * 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: osd client watch timeout = 120 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify cached snaps = True 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify missing on start = True 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: osd mclock override recovery settings = True 2024-03-29T11:41:28.679 INFO:teuthology.orchestra.run.smithi171.stdout: osd objectstore = bluestore 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: osd op queue = debug_random 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: osd op queue cut off = debug_random 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: osd shutdown pgref assert = True 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr] 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: debug mgr = 20 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: mgr/telemetry/nag = false 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout:[mon] 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-29T11:41:28.680 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: debug paxos = 20 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: mon data avail warn = 5 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: mon mgr mkfs grace = 240 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd reporter subtree level = osd 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd prime pg temp = true 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:41:28.681 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: # 1m isn't quite enough 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon min osdmap epochs = 50 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon osdmap full prune interval = 2 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon osdmap full prune min = 15 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon scrub interval = 300 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on pool no app = False 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: paxos service trim min = 10 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-29T11:41:28.682 INFO:teuthology.orchestra.run.smithi171.stdout:[client] 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout: rgw cache enabled = true 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable ops log = true 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable usage log = true 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout: debug monc = 20 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout: debug objecter = 20 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout: debug rados = 20 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.a] 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.c] 2024-03-29T11:41:28.683 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.b] 2024-03-29T11:41:28.685 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:41:28.685 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:41:28.740 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:41:28.740 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:28.740 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:41:28.755 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:28.755 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:41:28.810 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:41:28.810 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:28.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:41:28.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:41:28.890 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:41:28.890 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:28.945 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:41:28.945 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:41:28.945 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:41:28.945 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:41:28.985 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:41:28.985 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:29.039 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:41:29.040 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:41:29.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:41:29.094 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:41:29.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:41:29.172 INFO:teuthology.orchestra.run.smithi171.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:41:29.175 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:41:29.175 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:41:29.175 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:41:29.217 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:41:29.232 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:41:29.275 INFO:teuthology.orchestra.run.smithi171.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:41:29.290 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:41:29.291 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi171.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:41:29.291 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:41:29.317 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:29.317 INFO:tasks.ceph:role: osd.0 2024-03-29T11:41:29.317 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:41:29.317 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:41:29.411 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:41:29.411 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:41:29.411 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:41:29.411 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:41:29.411 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:41:29.411 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:41:29.411 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:41:29.411 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:41:29.412 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:41:29.412 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:41:30.728 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-29T11:41:30.731 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-29T11:41:30.731 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:41:30.768 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:41:30.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:41:30.912 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:30.912 INFO:tasks.ceph:role: osd.1 2024-03-29T11:41:30.912 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:41:30.912 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:41:31.008 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:41:31.008 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:41:31.008 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:41:31.008 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:41:31.008 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:41:31.009 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:41:31.009 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:41:31.009 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:41:31.009 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:41:31.009 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:41:32.349 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-29T11:41:32.351 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-29T11:41:32.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:41:32.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:41:32.464 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:41:32.534 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:32.534 INFO:tasks.ceph:role: osd.2 2024-03-29T11:41:32.534 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:41:32.534 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:41:32.631 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:41:32.631 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:41:32.631 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:41:32.631 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:41:32.631 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:41:32.631 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:41:32.632 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:41:32.632 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:41:32.632 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:41:32.632 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:41:33.973 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-29T11:41:33.976 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-29T11:41:33.976 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:41:34.013 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:41:34.087 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:41:34.156 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:34.156 INFO:tasks.ceph:role: osd.3 2024-03-29T11:41:34.156 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:41:34.156 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:41:34.252 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:41:34.252 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:41:34.252 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:41:34.252 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:41:34.252 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:41:34.252 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:41:34.252 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:41:34.253 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:41:34.253 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:41:34.253 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:41:35.607 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-29T11:41:35.610 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-29T11:41:35.610 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:41:35.647 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:41:35.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:35.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:35.817+0000 7fa579410640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:41:35.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:35.818+0000 7fa579410640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:41:35.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:35.818+0000 7fa579410640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:41:35.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:35.818+0000 7fa579410640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:41:35.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:35.818+0000 7fa579410640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:41:35.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:35.818+0000 7fa579410640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:41:38.707 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:41:38.739 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:38.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:38.829+0000 7f2f1e3fd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:41:38.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:38.830+0000 7f2f1e3fd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:41:38.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:38.830+0000 7f2f1e3fd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:41:38.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:38.830+0000 7f2f1e3fd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:41:38.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:38.830+0000 7f2f1e3fd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:41:38.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:38.831+0000 7f2f1e3fd640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:41:41.697 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:41:41.727 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:41.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:41.816+0000 7f677ec13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:41:41.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:41.816+0000 7f677ec13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:41:41.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:41.816+0000 7f677ec13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:41:41.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:41.816+0000 7f677ec13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:41:41.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:41.816+0000 7f677ec13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:41:41.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:41.817+0000 7f677ec13640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:41:44.706 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:41:44.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:44.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:44.826+0000 7faf62813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:41:44.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:44.826+0000 7faf62813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:41:44.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:44.826+0000 7faf62813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:41:44.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:44.826+0000 7faf62813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:41:44.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:44.826+0000 7faf62813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:41:44.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:41:44.827+0000 7faf62813640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:41:47.700 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:41:47.731 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:41:47.763 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:47.763 INFO:tasks.ceph:role: osd.4 2024-03-29T11:41:47.763 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:41:47.763 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:41:47.842 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:41:47.842 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:41:47.842 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:41:47.842 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:41:47.842 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:41:47.842 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:41:47.842 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:41:47.842 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:41:47.843 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:41:47.843 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:41:48.292 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-29T11:41:48.294 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-29T11:41:48.294 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:41:48.434 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:41:48.468 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:41:48.536 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:48.536 INFO:tasks.ceph:role: osd.5 2024-03-29T11:41:48.536 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:41:48.536 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:41:48.612 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:41:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:41:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:41:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:41:48.612 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:41:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:41:48.612 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:41:48.612 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:41:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:41:48.613 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:41:49.074 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-29T11:41:49.076 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-29T11:41:49.076 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:41:49.154 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:41:49.232 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:41:49.303 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:49.303 INFO:tasks.ceph:role: osd.6 2024-03-29T11:41:49.303 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:41:49.303 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:41:49.381 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:41:49.381 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:41:49.381 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:41:49.381 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:41:49.381 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:41:49.381 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:41:49.381 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:41:49.381 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:41:49.381 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:41:49.382 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:41:49.865 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-29T11:41:49.867 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-29T11:41:49.867 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:41:49.968 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:41:49.999 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:41:50.066 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:41:50.066 INFO:tasks.ceph:role: osd.7 2024-03-29T11:41:50.066 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:41:50.067 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:41:50.141 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:41:50.141 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:41:50.141 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:41:50.141 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:41:50.141 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:41:50.142 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:41:50.142 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:41:50.142 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:41:50.142 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:41:50.142 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:41:50.662 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-29T11:41:50.664 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-29T11:41:50.664 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:41:50.802 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:41:50.835 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:50.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:50.929+0000 7f846aeb5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:41:50.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:50.932+0000 7f846aeb5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:41:50.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:50.933+0000 7f846aeb5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:41:50.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:50.933+0000 7f846aeb5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:41:50.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:50.933+0000 7f846aeb5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:41:50.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:50.940+0000 7f846aeb5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:41:53.791 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:41:53.821 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:53.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:53.909+0000 7fc8de5aa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:41:53.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:53.909+0000 7fc8de5aa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:41:53.912 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:53.909+0000 7fc8de5aa640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:41:53.912 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:53.909+0000 7fc8de5aa640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:41:53.912 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:53.909+0000 7fc8de5aa640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:41:53.912 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:53.909+0000 7fc8de5aa640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:41:56.754 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:41:56.785 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:56.876 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:56.873+0000 7fd672e14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:41:56.876 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:56.874+0000 7fd672e14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:41:56.876 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:56.874+0000 7fd672e14640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:41:56.876 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:56.874+0000 7fd672e14640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:41:56.876 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:56.874+0000 7fd672e14640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:41:56.877 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:56.874+0000 7fd672e14640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:41:59.721 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:41:59.749 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:41:59.837 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:59.834+0000 7f03a90b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:41:59.837 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:59.834+0000 7f03a90b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:41:59.837 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:59.834+0000 7f03a90b5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:41:59.837 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:59.834+0000 7f03a90b5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:41:59.837 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:59.834+0000 7f03a90b5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:41:59.837 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:41:59.834+0000 7f03a90b5640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:42:02.678 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:42:02.708 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:42:02.708 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:02.708 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:42:02.740 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:02.740 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:42:02.812 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:02.812 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:42:02.882 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:02.882 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:42:02.951 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:02.951 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:42:03.021 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:03.021 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:42:03.050 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:03.050 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:42:03.119 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:03.119 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:42:03.187 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:03.188 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:42:03.255 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:03.255 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:42:03.323 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:03.323 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:42:03.341 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:03.341 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:42:03.378 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:42:03.378 DEBUG:teuthology.orchestra.run.smithi055:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:42:03.383 DEBUG:teuthology.orchestra.run.smithi171:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:42:03.421 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.y] 2024-03-29T11:42:03.421 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDpqAZm2dGOBRAAEozhCZHs2APKDK7YqjTrFw== 2024-03-29T11:42:03.421 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.0] 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDvqAZmau7OMBAA5IwAJZSDi2rHARv/A2Wm1g== 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.1] 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDyqAZmwRyKMRAAqPXS+h/7couR9SO+G9CvyA== 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.2] 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQD1qAZmdNu5MBAAtXYlNlwCYVorUrAbwW5r3g== 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.3] 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQD4qAZm9KNVMRAA3ULKYLxBXJ18qcVZgKjpBw== 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.x] 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDpqAZm8Qs9ChAAVIja4uTf65i4NZVUB/296A== 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.4] 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQD+qAZmKMByNxAA0rOLXB1Uc9aL27k5HIQajg== 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.5] 2024-03-29T11:42:03.422 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQABqQZm3ts8NhAAz+aol49aB3uR0umM5xNYOA== 2024-03-29T11:42:03.423 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.6] 2024-03-29T11:42:03.423 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAEqQZmSoQjNBAAbG1k8rEPB//ommiDJ32Qiw== 2024-03-29T11:42:03.423 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.7] 2024-03-29T11:42:03.423 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAHqQZmjeHEMRAAWj4Bb4WBdOqrUt0zsUxc6g== 2024-03-29T11:42:03.423 INFO:teuthology.orchestra.run.smithi055.stdout:[client.0] 2024-03-29T11:42:03.423 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDpqAZmmnTjDBAA5gTFb1X7zTMZsteIDkmB1w== 2024-03-29T11:42:03.423 INFO:teuthology.orchestra.run.smithi055.stdout:[client.1] 2024-03-29T11:42:03.423 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDpqAZmv+9cEBAAiwkPS3hJdYUCJH6tK7rcxw== 2024-03-29T11:42:03.443 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr.y] 2024-03-29T11:42:03.443 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDpqAZm2dGOBRAAEozhCZHs2APKDK7YqjTrFw== 2024-03-29T11:42:03.443 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.0] 2024-03-29T11:42:03.443 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDvqAZmau7OMBAA5IwAJZSDi2rHARv/A2Wm1g== 2024-03-29T11:42:03.443 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.1] 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDyqAZmwRyKMRAAqPXS+h/7couR9SO+G9CvyA== 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.2] 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQD1qAZmdNu5MBAAtXYlNlwCYVorUrAbwW5r3g== 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.3] 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQD4qAZm9KNVMRAA3ULKYLxBXJ18qcVZgKjpBw== 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr.x] 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDpqAZm8Qs9ChAAVIja4uTf65i4NZVUB/296A== 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.4] 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQD+qAZmKMByNxAA0rOLXB1Uc9aL27k5HIQajg== 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.5] 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQABqQZm3ts8NhAAz+aol49aB3uR0umM5xNYOA== 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.6] 2024-03-29T11:42:03.444 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQAEqQZmSoQjNBAAbG1k8rEPB//ommiDJ32Qiw== 2024-03-29T11:42:03.445 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.7] 2024-03-29T11:42:03.445 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQAHqQZmjeHEMRAAWj4Bb4WBdOqrUt0zsUxc6g== 2024-03-29T11:42:03.445 INFO:teuthology.orchestra.run.smithi171.stdout:[client.0] 2024-03-29T11:42:03.445 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDpqAZmmnTjDBAA5gTFb1X7zTMZsteIDkmB1w== 2024-03-29T11:42:03.445 INFO:teuthology.orchestra.run.smithi171.stdout:[client.1] 2024-03-29T11:42:03.445 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDpqAZmv+9cEBAAiwkPS3hJdYUCJH6tK7rcxw== 2024-03-29T11:42:03.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:42:03.467 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:42:03.538 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.540 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.595 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.646 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.649 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.701 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.703 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.754 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:42:03.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:42:03.808 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.810 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.862 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.918 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:03.971 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:42:04.022 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:42:04.024 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:42:04.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:42:04.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:42:04.136 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:42:04.136 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:42:04.163 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:42:04.259 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:42:04.285 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:42:04.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:42:04.449 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:42:04.475 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:42:04.503 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:42:04.601 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:42:04.628 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:42:04.630 DEBUG:teuthology.orchestra.run.smithi171:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:42:04.682 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:42:04.682 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:42:04.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:42:04.684 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:42:04.684 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:42:04.685 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:42:04.686 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:42:04.686 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:42:04.687 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:42:04.725 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:42:04.725 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:42:04.725 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:42:04.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:42:04.727 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:42:04.728 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:42:04.728 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:42:04.729 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:42:04.730 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:42:04.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.830+0000 7f3b42bd6640 10 monclient: get_monmap_and_config 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.830+0000 7f3b42bd6640 10 monclient: build_initial_monmap 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.830+0000 7f3b42bd6640 10 monclient: monmap: 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 0 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:42:04.830392+0000 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:42:04.830392+0000 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:04.833 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:04.833 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:04.833 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:04.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.830+0000 7f3b42bd6640 1 Processor -- start 2024-03-29T11:42:04.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.830+0000 7f3b42bd6640 1 -- start start 2024-03-29T11:42:04.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.830+0000 7f3b42bd6640 10 monclient: init 2024-03-29T11:42:04.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:04.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:04.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 10 monclient(hunting): picked mon.noname-c con 0x7f3b3c09ac40 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 10 monclient(hunting): picked mon.noname-b con 0x7f3b3c09c870 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 10 monclient(hunting): picked mon.noname-a con 0x7f3b3c0b4360 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c09dec0 con 0x7f3b3c0b4360 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c09da30 con 0x7f3b3c09ac40 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c0b39c0 con 0x7f3b3c09c870 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b42bd6640 10 monclient(hunting): authenticate will time out at 938.110207s 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b41173640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3b3c0b4360 0x7f3b3c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56862/0 (socket says 172.21.15.55:56862) 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b3bfff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f3b3c09c870 0x7f3b3c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:56606/0 (socket says 172.21.15.55:56606) 2024-03-29T11:42:04.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:04.831+0000 7f3b41173640 1 -- 172.21.15.55:0/2044363317 learned_addr learned my addr 172.21.15.55:0/2044363317 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:07.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 10 monclient(hunting): tick 2024-03-29T11:42:07.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:42:07.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 1 monclient(hunting): continuing hunt 2024-03-29T11:42:07.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:42:07.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.55:6790/0 conn(0x7f3b3c09c870 legacy=0x7f3b3c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:07.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.171:6789/0 conn(0x7f3b3c09ac40 legacy=0x7f3b3c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:07.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.55:6789/0 conn(0x7f3b3c0b4360 legacy=0x7f3b3c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:07.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:07.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f3b200036e0 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:07.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f3b200072d0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:07.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f3b2000afd0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:07.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c0b5160 con 0x7f3b2000afd0 2024-03-29T11:42:07.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2000fdd0 con 0x7f3b200036e0 2024-03-29T11:42:07.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b20011030 con 0x7f3b200072d0 2024-03-29T11:42:07.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:07.830+0000 7f3b3a7fc640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.041+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744993813 0 0) 0x7f3b3c0b5160 con 0x7f3b2000afd0 2024-03-29T11:42:10.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.041+0000 7f3b3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.041+0000 7f3b3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.041+0000 7f3b3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.041+0000 7f3b3b7fe640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:42:10.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.041+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2c003520 con 0x7f3b2000afd0 2024-03-29T11:42:10.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803752893 0 0) 0x7f3b2000fdd0 con 0x7f3b200036e0 2024-03-29T11:42:10.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:42:10.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3c0b5160 con 0x7f3b200036e0 2024-03-29T11:42:10.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108690540 0 0) 0x7f3b20011030 con 0x7f3b200072d0 2024-03-29T11:42:10.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:42:10.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2000fdd0 con 0x7f3b200072d0 2024-03-29T11:42:10.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1406191471 0 0) 0x7f3b3c0b5160 con 0x7f3b200036e0 2024-03-29T11:42:10.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.55:6790/0 conn(0x7f3b200072d0 legacy=0x7f3b20009740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.55:6789/0 conn(0x7f3b2000afd0 legacy=0x7f3b2000d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:42:10.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20012290 con 0x7f3b200036e0 2024-03-29T11:42:10.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3b30001610 con 0x7f3b200036e0 2024-03-29T11:42:10.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:10.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b3b7fe640 10 monclient: dump: 2024-03-29T11:42:10.056 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:10.056 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.058 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:10.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b42bd6640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:42:10.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.042+0000 7f3b42bd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:10.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b40972640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.171:6789/0 conn(0x7f3b200036e0 legacy=0x7f3b20005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:42:10.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b40972640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.171:6789/0 conn(0x7f3b200036e0 legacy=0x7f3b20005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:42:10.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b40972640 1 --1- 172.21.15.55:0/2044363317 >> v1:172.21.15.171:6789/0 conn(0x7f3b200036e0 0x7f3b20005b30 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T11:42:10.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b40972640 1 --1- 172.21.15.55:0/2044363317 >> v1:172.21.15.171:6789/0 conn(0x7f3b200036e0 0x7f3b20005b30 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:42:10.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.171:6789/0 2024-03-29T11:42:10.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.171:6789/0 conn(0x7f3b200036e0 legacy=0x7f3b20005b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:42:10.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:10.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient(hunting): picked mon.b con 0x7f3b2c003e80 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:10.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient(hunting): picked mon.a con 0x7f3b2000afd0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:10.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient(hunting): picked mon.c con 0x7f3b200072d0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:10.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3b3c0b5160 con 0x7f3b2000afd0 2024-03-29T11:42:10.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3b20011030 con 0x7f3b2c003e80 2024-03-29T11:42:10.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3b2c004590 con 0x7f3b200072d0 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.171:6789/0 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243303686 0 0) 0x7f3b3c0b5160 con 0x7f3b2000afd0 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.55:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f3b2c00f0c0 con 0x7f3b2000afd0 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1481592787 0 0) 0x7f3b2c00f0c0 con 0x7f3b2000afd0 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.55:6790/0 conn(0x7f3b200072d0 legacy=0x7f3b2c00d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.171:6789/0 conn(0x7f3b2c003e80 legacy=0x7f3b2c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 monclient: found mon.a 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f3b2c012250 con 0x7f3b2000afd0 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3b28001730 con 0x7f3b2000afd0 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b3b7fe640 10 monclient: dump: 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.063 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.063 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.063 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.063 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:10.063 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.063 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.063 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.063 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:10.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.043+0000 7f3b42bd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:10.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.044+0000 7f3b3b7fe640 1 -- 172.21.15.55:0/2044363317 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b28001890 con 0x7f3b2000afd0 2024-03-29T11:42:10.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.044+0000 7f3b3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.044+0000 7f3b42bd6640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.044+0000 7f3b42bd6640 4 set_mon_vals no callback set 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.044+0000 7f3b42bd6640 10 monclient: shutdown 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.044+0000 7f3b42bd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.044+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2044363317 >> v1:172.21.15.55:6789/0 conn(0x7f3b2000afd0 legacy=0x7f3b2c004ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.044+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2044363317 shutdown_connections 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.044+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2044363317 >> 172.21.15.55:0/2044363317 conn(0x7f3b3c066b00 msgr2=0x7f3b3c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.045+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2044363317 shutdown_connections 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.045+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2044363317 wait complete. 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.046+0000 7f3b42bd6640 10 monclient: build_initial_monmap 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.046+0000 7f3b42bd6640 10 monclient: monmap: 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 0 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 0.000000 2024-03-29T11:42:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:created 0.000000 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.046+0000 7f3b42bd6640 1 librados: starting msgr at 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.046+0000 7f3b42bd6640 1 librados: starting objecter 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.046+0000 7f3b42bd6640 1 Processor -- start 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.046+0000 7f3b42bd6640 1 -- start start 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.047+0000 7f3b42bd6640 1 librados: setting wanted keys 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.047+0000 7f3b42bd6640 1 librados: calling monclient init 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.047+0000 7f3b42bd6640 10 monclient: init 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.047+0000 7f3b42bd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b42bd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:10.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b42bd6640 10 monclient(hunting): picked mon.noname-a con 0x7f3b200072d0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b42bd6640 10 monclient(hunting): picked mon.noname-b con 0x7f3b2000afd0 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b42bd6640 10 monclient(hunting): picked mon.noname-c con 0x7f3b2c003e80 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b42bd6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c144be0 con 0x7f3b200072d0 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b42bd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c143d20 con 0x7f3b2000afd0 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b40972640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3b200072d0 0x7f3b3c1daa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45900/0 (socket says 172.21.15.55:45900) 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b40972640 1 -- 172.21.15.55:0/2822397411 learned_addr learned my addr 172.21.15.55:0/2822397411 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c1da520 con 0x7f3b2c003e80 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b42bd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b42bd6640 10 monclient(hunting): authenticate will time out at 943.327889s 2024-03-29T11:42:10.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 client.?.objecter ms_handle_connect 0x7f3b200072d0 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 client.?.objecter ms_handle_connect 0x7f3b2000afd0 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 client.?.objecter ms_handle_connect 0x7f3b2c003e80 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386266994 0 0) 0x7f3b3c144be0 con 0x7f3b200072d0 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.048+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b200037d0 con 0x7f3b200072d0 2024-03-29T11:42:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2515393308 0 0) 0x7f3b200037d0 con 0x7f3b200072d0 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b3c144be0 con 0x7f3b200072d0 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3b30001ab0 con 0x7f3b200072d0 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient(hunting): dump: 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.074 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 748221185 0 0) 0x7f3b3c144be0 con 0x7f3b200072d0 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 >> v1:172.21.15.55:6790/0 conn(0x7f3b2c003e80 legacy=0x7f3b3c1e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 >> v1:172.21.15.171:6789/0 conn(0x7f3b2000afd0 legacy=0x7f3b3c145a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 monclient: found mon.a 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c1db150 con 0x7f3b200072d0 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b42bd6640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:42:10.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b42bd6640 10 monclient: _renew_subs 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b42bd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3c1452e0 con 0x7f3b200072d0 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b42bd6640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b42bd6640 10 monclient: _renew_subs 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b42bd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3c1438f0 con 0x7f3b200072d0 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b42bd6640 1 librados: init done 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b30001c90 con 0x7f3b200072d0 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3b30003a40 con 0x7f3b200072d0 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b3affd640 4 set_mon_vals no callback set 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 monclient: dump: 2024-03-29T11:42:10.076 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3b30003fb0 con 0x7f3b200072d0 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3041953726 0 0) 0x7f3b300065a0 con 0x7f3b200072d0 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 10 client.4110.objecter ms_dispatch 0x7f3b3c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.049+0000 7f3b397fa640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:42:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.050+0000 7f3b397fa640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:42:10.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.050+0000 7f3b42bd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.050+0000 7f3b42bd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.050+0000 7f3b42bd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:10.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.050+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c005080 con 0x7f3b200072d0 2024-03-29T11:42:10.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.054+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3b300067b0 con 0x7f3b200072d0 2024-03-29T11:42:10.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.054+0000 7f3b397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.054+0000 7f3b397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:10.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.125+0000 7f3b42bd6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:42:10.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.125+0000 7f3b42bd6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:42:10.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.125+0000 7f3b42bd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:10.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.125+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3b1c005330 con 0x7f3b200072d0 2024-03-29T11:42:10.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.126+0000 7f3b397fa640 1 -- 172.21.15.55:0/2822397411 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f3b300058a0 con 0x7f3b200072d0 2024-03-29T11:42:10.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.127+0000 7f3b397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:42:10.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.127+0000 7f3b397fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:42:10.130 INFO:teuthology.orchestra.run.smithi055.stderr:adjusted tunables profile to default 2024-03-29T11:42:10.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 10 librados: watch_flush enter 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 10 librados: watch_flush exit 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 10 monclient: shutdown 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 >> v1:172.21.15.55:6789/0 conn(0x7f3b200072d0 legacy=0x7f3b3c1daa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 shutdown_connections 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 >> 172.21.15.55:0/2822397411 conn(0x7f3b3c066b00 msgr2=0x7f3b3c13be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 shutdown_connections 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 1 -- 172.21.15.55:0/2822397411 wait complete. 2024-03-29T11:42:10.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:10.128+0000 7f3b42bd6640 1 librados: shutdown 2024-03-29T11:42:10.143 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:42:10.143 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:42:10.143 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:10.143 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:42:10.173 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:10.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:42:10.245 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:10.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:42:10.315 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-29T11:42:10.315 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:42:10.383 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:10.383 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:42:10.411 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:10.411 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:42:10.478 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:10.478 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:42:10.546 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-29T11:42:10.546 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:42:10.555 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:10.552+0000 7fa3002c5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:42:10.560 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:10.558+0000 7f9d20c32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:42:10.587 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 64e3cee1-493f-490b-8c64-884ddab30d9d 0 2024-03-29T11:42:10.611 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:10.608+0000 7fa3002c5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:42:10.617 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:10.615+0000 7f9d20c32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.714+0000 7f56f43b5640 10 monclient: get_monmap_and_config 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.714+0000 7f56f43b5640 10 monclient: build_initial_monmap 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.714+0000 7f56f43b5640 10 monclient: monmap: 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:42:10.715457+0000 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:42:10.715457+0000 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.717 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:10.718 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:10.718 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:10.718 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 1 Processor -- start 2024-03-29T11:42:10.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 1 -- start start 2024-03-29T11:42:10.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 10 monclient: init 2024-03-29T11:42:10.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:10.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 10 monclient(hunting): picked mon.noname-b con 0x7f56ec09ac40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:10.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 10 monclient(hunting): picked mon.noname-c con 0x7f56ec09c870 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 10 monclient(hunting): picked mon.noname-a con 0x7f56ec0b4360 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec09dec0 con 0x7f56ec0b4360 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec09da30 con 0x7f56ec09c870 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec0b39c0 con 0x7f56ec09ac40 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f43b5640 10 monclient(hunting): authenticate will time out at 935.015796s 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.715+0000 7f56f1950640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f56ec09c870 0x7f56ec13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42734/0 (socket says 172.21.15.171:42734) 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f1950640 1 -- 172.21.15.171:0/2858638452 learned_addr learned my addr 172.21.15.171:0/2858638452 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077767186 0 0) 0x7f56ec09da30 con 0x7f56ec09c870 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:42:10.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d8003520 con 0x7f56ec09c870 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562938873 0 0) 0x7f56e4001490 con 0x7f56ec0b4360 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56ec09da30 con 0x7f56ec0b4360 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654954377 0 0) 0x7f56ec0b39c0 con 0x7f56ec09ac40 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e4001490 con 0x7f56ec09ac40 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728879411 0 0) 0x7f56d8003520 con 0x7f56ec09c870 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 >> v1:172.21.15.55:6790/0 conn(0x7f56ec09ac40 legacy=0x7f56ec0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 >> v1:172.21.15.55:6789/0 conn(0x7f56ec0b4360 legacy=0x7f56ec14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 monclient: found mon.noname-c 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec0b5160 con 0x7f56ec09c870 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f43b5640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f56e8001690 con 0x7f56ec09c870 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f43b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: dump: 2024-03-29T11:42:10.730 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f43b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56e8001870 con 0x7f56ec09c870 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f43b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 1 -- 172.21.15.171:0/2858638452 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f56e8001c80 con 0x7f56ec09c870 2024-03-29T11:42:10.731 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f43b5640 4 set_mon_vals no callback set 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.716+0000 7f56f114f640 10 monclient: dump: 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.717+0000 7f56f43b5640 10 monclient: shutdown 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.717+0000 7f56f43b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:10.732 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.717+0000 7f56f43b5640 1 -- 172.21.15.171:0/2858638452 >> v1:172.21.15.171:6789/0 conn(0x7f56ec09c870 legacy=0x7f56ec13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.717+0000 7f56f43b5640 1 -- 172.21.15.171:0/2858638452 shutdown_connections 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.717+0000 7f56f43b5640 1 -- 172.21.15.171:0/2858638452 >> 172.21.15.171:0/2858638452 conn(0x7f56ec066b00 msgr2=0x7f56ec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.717+0000 7f56f43b5640 1 -- 172.21.15.171:0/2858638452 shutdown_connections 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.717+0000 7f56f43b5640 1 -- 172.21.15.171:0/2858638452 wait complete. 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.717+0000 7f56f43b5640 10 monclient: build_initial_monmap 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.717+0000 7f56f43b5640 10 monclient: monmap: 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:10.733 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 1 librados: starting msgr at 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 1 librados: starting objecter 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 1 Processor -- start 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 1 -- start start 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 1 librados: setting wanted keys 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 1 librados: calling monclient init 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 10 monclient: init 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 10 monclient(hunting): picked mon.noname-c con 0x7f56ec09ac40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 10 monclient(hunting): picked mon.noname-b con 0x7f56ec0b6b00 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 10 monclient(hunting): picked mon.noname-a con 0x7f56ec1406b0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:10.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec1d7e20 con 0x7f56ec1406b0 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec140b00 con 0x7f56ec0b6b00 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec0ad080 con 0x7f56ec09ac40 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f43b5640 10 monclient(hunting): authenticate will time out at 935.018769s 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f1950640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f56ec0b6b00 0x7f56ec13ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42738/0 (socket says 172.21.15.171:42738) 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56f1950640 1 -- 172.21.15.171:0/635129165 learned_addr learned my addr 172.21.15.171:0/635129165 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.718+0000 7f56e2ffd640 10 client.?.objecter ms_handle_connect 0x7f56ec0b6b00 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 client.?.objecter ms_handle_connect 0x7f56ec09ac40 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 client.?.objecter ms_handle_connect 0x7f56ec1406b0 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227166819 0 0) 0x7f56ec140b00 con 0x7f56ec0b6b00 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.735 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d0004eb0 con 0x7f56ec0b6b00 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643396580 0 0) 0x7f56ec0ad080 con 0x7f56ec09ac40 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56ec140b00 con 0x7f56ec09ac40 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066131939 0 0) 0x7f56ec1d7e20 con 0x7f56ec1406b0 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:42:10.736 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56ec0ad080 con 0x7f56ec1406b0 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 117320081 0 0) 0x7f56d0004eb0 con 0x7f56ec0b6b00 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56ec1d7e20 con 0x7f56ec0b6b00 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f56e8001bc0 con 0x7f56ec0b6b00 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.737 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1788140950 0 0) 0x7f56ec140b00 con 0x7f56ec09ac40 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56d0004eb0 con 0x7f56ec09ac40 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f56dc001820 con 0x7f56ec09ac40 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2165851265 0 0) 0x7f56ec0ad080 con 0x7f56ec1406b0 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56ec140b00 con 0x7f56ec1406b0 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f56e4001a00 con 0x7f56ec1406b0 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:10.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4177493079 0 0) 0x7f56ec1d7e20 con 0x7f56ec0b6b00 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 >> v1:172.21.15.55:6790/0 conn(0x7f56ec09ac40 legacy=0x7f56ec1d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 >> v1:172.21.15.55:6789/0 conn(0x7f56ec1406b0 legacy=0x7f56ec1e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.740 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 monclient: found mon.b 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec1d7ab0 con 0x7f56ec0b6b00 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f43b5640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f43b5640 10 monclient: _renew_subs 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f43b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f43b5640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ec1e3570 con 0x7f56ec0b6b00 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f43b5640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f43b5640 10 monclient: _renew_subs 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f43b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f43b5640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec1e38e0 con 0x7f56ec0b6b00 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f43b5640 1 librados: init done 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56e8004bf0 con 0x7f56ec0b6b00 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f56e8004f00 con 0x7f56ec0b6b00 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56f094e640 4 set_mon_vals no callback set 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 monclient: dump: 2024-03-29T11:42:10.742 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f56e80053f0 con 0x7f56ec0b6b00 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3572809768 0 0) 0x7f56e80067d0 con 0x7f56ec0b6b00 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 10 client.4114.objecter ms_dispatch 0x7f56ec000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:42:10.743 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.719+0000 7f56e2ffd640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:42:10.744 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.720+0000 7f56f43b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.744 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.720+0000 7f56f43b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.744 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.720+0000 7f56f43b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.744 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.720+0000 7f56f43b5640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d4005080 con 0x7f56ec0b6b00 2024-03-29T11:42:10.744 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.722+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f56e8006990 con 0x7f56ec0b6b00 2024-03-29T11:42:10.744 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.722+0000 7f56e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.744 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.722+0000 7f56e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:10.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.795+0000 7f56f43b5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "64e3cee1-493f-490b-8c64-884ddab30d9d", "id": 0}] 2024-03-29T11:42:10.798 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.795+0000 7f56f43b5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "64e3cee1-493f-490b-8c64-884ddab30d9d", "id": 0}] 2024-03-29T11:42:10.798 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.795+0000 7f56f43b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.798 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.795+0000 7f56f43b5640 1 -- 172.21.15.171:0/635129165 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "64e3cee1-493f-490b-8c64-884ddab30d9d", "id": 0} v 0) v1 -- 0x7f56d40058b0 con 0x7f56ec0b6b00 2024-03-29T11:42:10.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.802+0000 7f56e2ffd640 1 -- 172.21.15.171:0/635129165 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "64e3cee1-493f-490b-8c64-884ddab30d9d", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 265669080 0 732707515) 0x7f56e8005860 con 0x7f56ec0b6b00 2024-03-29T11:42:10.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.802+0000 7f56e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "64e3cee1-493f-490b-8c64-884ddab30d9d", "id": 0}] 2024-03-29T11:42:10.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.802+0000 7f56e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:10.805 INFO:teuthology.orchestra.run.smithi171.stdout:0 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 10 librados: watch_flush enter 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 10 librados: watch_flush exit 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 10 monclient: shutdown 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 1 -- 172.21.15.171:0/635129165 >> v1:172.21.15.171:6789/0 conn(0x7f56ec0b6b00 legacy=0x7f56ec13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 1 -- 172.21.15.171:0/635129165 shutdown_connections 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 1 -- 172.21.15.171:0/635129165 >> 172.21.15.171:0/635129165 conn(0x7f56ec066b00 msgr2=0x7f56ec0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 1 -- 172.21.15.171:0/635129165 shutdown_connections 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 1 -- 172.21.15.171:0/635129165 wait complete. 2024-03-29T11:42:10.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.803+0000 7f56f43b5640 1 librados: shutdown 2024-03-29T11:42:10.816 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new b8c09668-0d5e-4a37-bac5-dcfaaa0e3677 1 2024-03-29T11:42:10.918 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.916+0000 7f719e1c6640 10 monclient: get_monmap_and_config 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.916+0000 7f719e1c6640 10 monclient: build_initial_monmap 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.916+0000 7f719e1c6640 10 monclient: monmap: 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:42:10.917191+0000 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:42:10.917191+0000 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:10.919 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.916+0000 7f719e1c6640 1 Processor -- start 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 1 -- start start 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 10 monclient: init 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f719809ac40 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f719809c870 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f71980b4360 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f719809dec0 con 0x7f719809c870 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f719809da30 con 0x7f719809ac40 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71980b39c0 con 0x7f71980b4360 2024-03-29T11:42:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f719e1c6640 10 monclient(hunting): authenticate will time out at 935.217513s 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f71977fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f719809ac40 0x7f71980b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42754/0 (socket says 172.21.15.171:42754) 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f71977fe640 1 -- 172.21.15.171:0/178645443 learned_addr learned my addr 172.21.15.171:0/178645443 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279408779 0 0) 0x7f719809da30 con 0x7f719809ac40 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f71967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f71967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f71967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f71967fc640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.917+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7180003520 con 0x7f719809ac40 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051369609 0 0) 0x7f71980b39c0 con 0x7f71980b4360 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:42:10.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f719809da30 con 0x7f71980b4360 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677588055 0 0) 0x7f7188001110 con 0x7f719809c870 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71980b39c0 con 0x7f719809c870 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362155559 0 0) 0x7f7180003520 con 0x7f719809ac40 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 >> v1:172.21.15.55:6790/0 conn(0x7f71980b4360 legacy=0x7f719814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 >> v1:172.21.15.55:6789/0 conn(0x7f719809c870 legacy=0x7f719813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 monclient: found mon.noname-c 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71980b5160 con 0x7f719809ac40 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f718c001690 con 0x7f719809ac40 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: dump: 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.931 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f718c001870 con 0x7f719809ac40 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 4 set_mon_vals no callback set 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 1 -- 172.21.15.171:0/178645443 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f718c001c80 con 0x7f719809ac40 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f71967fc640 10 monclient: dump: 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.932 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 10 monclient: shutdown 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 1 -- 172.21.15.171:0/178645443 >> v1:172.21.15.171:6789/0 conn(0x7f719809ac40 legacy=0x7f71980b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 1 -- 172.21.15.171:0/178645443 shutdown_connections 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 1 -- 172.21.15.171:0/178645443 >> 172.21.15.171:0/178645443 conn(0x7f7198066b00 msgr2=0x7f7198066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:10.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 1 -- 172.21.15.171:0/178645443 shutdown_connections 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.918+0000 7f719e1c6640 1 -- 172.21.15.171:0/178645443 wait complete. 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.919+0000 7f719e1c6640 10 monclient: build_initial_monmap 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.919+0000 7f719e1c6640 10 monclient: monmap: 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:10.934 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.919+0000 7f719e1c6640 1 librados: starting msgr at 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.919+0000 7f719e1c6640 1 librados: starting objecter 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.919+0000 7f719e1c6640 1 Processor -- start 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.919+0000 7f719e1c6640 1 -- start start 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.919+0000 7f719e1c6640 1 librados: setting wanted keys 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.919+0000 7f719e1c6640 1 librados: calling monclient init 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.919+0000 7f719e1c6640 10 monclient: init 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.935 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f719809ac40 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f71980adbe0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f7198136a40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71980ae720 con 0x7f71980adbe0 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71980af7a0 con 0x7f719809ac40 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71980ae200 con 0x7f7198136a40 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f719e1c6640 10 monclient(hunting): authenticate will time out at 935.220348s 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f71977fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f719809ac40 0x7f71980ad3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42768/0 (socket says 172.21.15.171:42768) 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f71977fe640 1 -- 172.21.15.171:0/3685600659 learned_addr learned my addr 172.21.15.171:0/3685600659 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 client.?.objecter ms_handle_connect 0x7f719809ac40 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 client.?.objecter ms_handle_connect 0x7f71980adbe0 2024-03-29T11:42:10.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 client.?.objecter ms_handle_connect 0x7f7198136a40 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57368061 0 0) 0x7f71980af7a0 con 0x7f719809ac40 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7184004eb0 con 0x7f719809ac40 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127554068 0 0) 0x7f71980ae720 con 0x7f71980adbe0 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71980af7a0 con 0x7f71980adbe0 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833364061 0 0) 0x7f71980ae200 con 0x7f7198136a40 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71980ae720 con 0x7f7198136a40 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1409608367 0 0) 0x7f7184004eb0 con 0x7f719809ac40 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71980ae200 con 0x7f719809ac40 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f718c001bc0 con 0x7f719809ac40 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:10.938 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): dump: 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:10.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.920+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 94429986 0 0) 0x7f71980af7a0 con 0x7f71980adbe0 2024-03-29T11:42:10.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7184004eb0 con 0x7f71980adbe0 2024-03-29T11:42:10.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7188001860 con 0x7f71980adbe0 2024-03-29T11:42:10.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:10.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient(hunting): dump: 2024-03-29T11:42:10.940 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.940 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.940 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.940 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2157653538 0 0) 0x7f71980ae720 con 0x7f7198136a40 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71980af7a0 con 0x7f7198136a40 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f71900019f0 con 0x7f7198136a40 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient(hunting): dump: 2024-03-29T11:42:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3242070164 0 0) 0x7f71980ae200 con 0x7f719809ac40 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 >> v1:172.21.15.55:6790/0 conn(0x7f7198136a40 legacy=0x7f71980b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 >> v1:172.21.15.55:6789/0 conn(0x7f71980adbe0 legacy=0x7f7198136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 monclient: found mon.b 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71980b6c20 con 0x7f719809ac40 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f719e1c6640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f719e1c6640 10 monclient: _renew_subs 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f719e1c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f719e1c6640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71980b6ef0 con 0x7f719809ac40 2024-03-29T11:42:10.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f719e1c6640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f719e1c6640 10 monclient: _renew_subs 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f719e1c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f719e1c6640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71981e3d10 con 0x7f719809ac40 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f719e1c6640 1 librados: init done 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f718c004bf0 con 0x7f719809ac40 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f718c004f00 con 0x7f719809ac40 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f7195ffb640 4 set_mon_vals no callback set 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 monclient: dump: 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:10.944 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f718c0053f0 con 0x7f719809ac40 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 898992077 0 0) 0x7f718c0068e0 con 0x7f719809ac40 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 10 client.4120.objecter ms_dispatch 0x7f7198000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.921+0000 7f717ffff640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.922+0000 7f719e1c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.922+0000 7f719e1c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.946 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.922+0000 7f719e1c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.946 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.922+0000 7f719e1c6640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7178005080 con 0x7f719809ac40 2024-03-29T11:42:10.946 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.924+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f718c006bc0 con 0x7f719809ac40 2024-03-29T11:42:10.946 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.924+0000 7f717ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.946 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.924+0000 7f717ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:10.998 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.996+0000 7f719e1c6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b8c09668-0d5e-4a37-bac5-dcfaaa0e3677", "id": 1}] 2024-03-29T11:42:10.999 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.996+0000 7f719e1c6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b8c09668-0d5e-4a37-bac5-dcfaaa0e3677", "id": 1}] 2024-03-29T11:42:10.999 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.996+0000 7f719e1c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:10.999 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:10.996+0000 7f719e1c6640 1 -- 172.21.15.171:0/3685600659 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b8c09668-0d5e-4a37-bac5-dcfaaa0e3677", "id": 1} v 0) v1 -- 0x7f71780058b0 con 0x7f719809ac40 2024-03-29T11:42:11.013 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.010+0000 7f717ffff640 1 -- 172.21.15.171:0/3685600659 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b8c09668-0d5e-4a37-bac5-dcfaaa0e3677", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4199845558 0 940483276) 0x7f718c005910 con 0x7f719809ac40 2024-03-29T11:42:11.013 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.010+0000 7f717ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b8c09668-0d5e-4a37-bac5-dcfaaa0e3677", "id": 1}] 2024-03-29T11:42:11.013 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.010+0000 7f717ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:11.014 INFO:teuthology.orchestra.run.smithi171.stdout:1 2024-03-29T11:42:11.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 10 librados: watch_flush enter 2024-03-29T11:42:11.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 10 librados: watch_flush exit 2024-03-29T11:42:11.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:42:11.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:42:11.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 10 monclient: shutdown 2024-03-29T11:42:11.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 1 -- 172.21.15.171:0/3685600659 >> v1:172.21.15.171:6789/0 conn(0x7f719809ac40 legacy=0x7f71980ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 1 -- 172.21.15.171:0/3685600659 shutdown_connections 2024-03-29T11:42:11.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 1 -- 172.21.15.171:0/3685600659 >> 172.21.15.171:0/3685600659 conn(0x7f7198066b00 msgr2=0x7f719813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.011+0000 7f719e1c6640 1 -- 172.21.15.171:0/3685600659 shutdown_connections 2024-03-29T11:42:11.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.012+0000 7f719e1c6640 1 -- 172.21.15.171:0/3685600659 wait complete. 2024-03-29T11:42:11.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.012+0000 7f719e1c6640 1 librados: shutdown 2024-03-29T11:42:11.025 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 49182b31-63d6-416d-a58c-71143c8d99f2 2 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.123+0000 7fe7ed732640 10 monclient: get_monmap_and_config 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.123+0000 7fe7ed732640 10 monclient: build_initial_monmap 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.123+0000 7fe7ed732640 10 monclient: monmap: 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:42:11.124403+0000 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:42:11.124403+0000 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:11.126 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:11.127 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:11.127 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 1 Processor -- start 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 1 -- start start 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 10 monclient: init 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 10 monclient(hunting): picked mon.noname-c con 0x7fe7e80b8240 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 10 monclient(hunting): picked mon.noname-b con 0x7fe7e80b1f60 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 10 monclient(hunting): picked mon.noname-a con 0x7fe7e809c8c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e809bd80 con 0x7fe7e809c8c0 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e809b960 con 0x7fe7e80b8240 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80b2ec0 con 0x7fe7e80b1f60 2024-03-29T11:42:11.136 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7ed732640 10 monclient(hunting): authenticate will time out at 935.424718s 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7e6ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe7e80b8240 0x7fe7e813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42774/0 (socket says 172.21.15.171:42774) 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.124+0000 7fe7e6ffd640 1 -- 172.21.15.171:0/4034431718 learned_addr learned my addr 172.21.15.171:0/4034431718 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395397006 0 0) 0x7fe7e809b960 con 0x7fe7e80b8240 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d0003520 con 0x7fe7e80b8240 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381230458 0 0) 0x7fe7e809bd80 con 0x7fe7e809c8c0 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:42:11.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e809b960 con 0x7fe7e809c8c0 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566801050 0 0) 0x7fe7e80b2ec0 con 0x7fe7e80b1f60 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e809bd80 con 0x7fe7e80b1f60 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1676955869 0 0) 0x7fe7d0003520 con 0x7fe7e80b8240 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 >> v1:172.21.15.55:6790/0 conn(0x7fe7e80b1f60 legacy=0x7fe7e809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 >> v1:172.21.15.55:6789/0 conn(0x7fe7e809c8c0 legacy=0x7fe7e814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:11.138 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e813fb60 con 0x7fe7e80b8240 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe7d8001690 con 0x7fe7e80b8240 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7ed732640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7ed732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: dump: 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.139 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7ed732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7d8001870 con 0x7fe7e80b8240 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7ed732640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 1 -- 172.21.15.171:0/4034431718 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe7d8001c80 con 0x7fe7e80b8240 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7ed732640 4 set_mon_vals no callback set 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.125+0000 7fe7e5ffb640 10 monclient: dump: 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.140 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 10 monclient: shutdown 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.141 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 1 -- 172.21.15.171:0/4034431718 >> v1:172.21.15.171:6789/0 conn(0x7fe7e80b8240 legacy=0x7fe7e813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 1 -- 172.21.15.171:0/4034431718 shutdown_connections 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 1 -- 172.21.15.171:0/4034431718 >> 172.21.15.171:0/4034431718 conn(0x7fe7e8066b00 msgr2=0x7fe7e8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 1 -- 172.21.15.171:0/4034431718 shutdown_connections 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 1 -- 172.21.15.171:0/4034431718 wait complete. 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 10 monclient: build_initial_monmap 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 10 monclient: monmap: 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:11.142 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 1 librados: starting msgr at 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.126+0000 7fe7ed732640 1 librados: starting objecter 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 1 Processor -- start 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 1 -- start start 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 1 librados: setting wanted keys 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 1 librados: calling monclient init 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 10 monclient: init 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 10 monclient(hunting): picked mon.noname-a con 0x7fe7e809c8c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 10 monclient(hunting): picked mon.noname-b con 0x7fe7e80b8240 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 10 monclient(hunting): picked mon.noname-c con 0x7fe7e81d7ac0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:11.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e8135830 con 0x7fe7e809c8c0 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80b08d0 con 0x7fe7e80b8240 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80af1b0 con 0x7fe7e81d7ac0 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.127+0000 7fe7ed732640 10 monclient(hunting): authenticate will time out at 935.427871s 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7e67fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe7e80b8240 0x7fe7e8134a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42776/0 (socket says 172.21.15.171:42776) 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7e67fc640 1 -- 172.21.15.171:0/2290188523 learned_addr learned my addr 172.21.15.171:0/2290188523 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 client.?.objecter ms_handle_connect 0x7fe7e80b8240 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 client.?.objecter ms_handle_connect 0x7fe7e809c8c0 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 client.?.objecter ms_handle_connect 0x7fe7e81d7ac0 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240459177 0 0) 0x7fe7e80b08d0 con 0x7fe7e80b8240 2024-03-29T11:42:11.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d4004eb0 con 0x7fe7e80b8240 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541323372 0 0) 0x7fe7e8135830 con 0x7fe7e809c8c0 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e80b08d0 con 0x7fe7e809c8c0 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716722432 0 0) 0x7fe7e80af1b0 con 0x7fe7e81d7ac0 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e8135830 con 0x7fe7e81d7ac0 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1185790305 0 0) 0x7fe7d4004eb0 con 0x7fe7e80b8240 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7e80af1b0 con 0x7fe7e80b8240 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe7dc001910 con 0x7fe7e80b8240 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.146 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3270428490 0 0) 0x7fe7e80b08d0 con 0x7fe7e809c8c0 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7d4004eb0 con 0x7fe7e809c8c0 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 678363405 0 0) 0x7fe7e8135830 con 0x7fe7e81d7ac0 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7e80b08d0 con 0x7fe7e81d7ac0 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe7d8001b80 con 0x7fe7e809c8c0 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.147 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe7e0001c30 con 0x7fe7e81d7ac0 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.148 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4009711816 0 0) 0x7fe7e80af1b0 con 0x7fe7e80b8240 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 >> v1:172.21.15.55:6790/0 conn(0x7fe7e81d7ac0 legacy=0x7fe7e8135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 >> v1:172.21.15.55:6789/0 conn(0x7fe7e809c8c0 legacy=0x7fe7e81d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 monclient: found mon.b 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e80af6d0 con 0x7fe7e80b8240 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:11.149 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7ed732640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7ed732640 10 monclient: _renew_subs 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7ed732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7ed732640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e80aedd0 con 0x7fe7e80b8240 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7ed732640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7ed732640 10 monclient: _renew_subs 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7ed732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7ed732640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e80aea50 con 0x7fe7e80b8240 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7ed732640 1 librados: init done 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7dc001a70 con 0x7fe7e80b8240 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe7dc003830 con 0x7fe7e80b8240 2024-03-29T11:42:11.150 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7e57fa640 4 set_mon_vals no callback set 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 10 monclient: dump: 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.128+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe7dc003de0 con 0x7fe7e80b8240 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.129+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3921987565 0 0) 0x7fe7dc0065a0 con 0x7fe7e80b8240 2024-03-29T11:42:11.151 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.129+0000 7fe7cf7fe640 10 client.4126.objecter ms_dispatch 0x7fe7e8000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.129+0000 7fe7cf7fe640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.129+0000 7fe7cf7fe640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.129+0000 7fe7cf7fe640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.129+0000 7fe7ed732640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.129+0000 7fe7ed732640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.129+0000 7fe7ed732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.129+0000 7fe7ed732640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c8005080 con 0x7fe7e80b8240 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.131+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe7dc0043d0 con 0x7fe7e80b8240 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.131+0000 7fe7cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.152 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.131+0000 7fe7cf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:11.172 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.169+0000 7fa3002c5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:42:11.178 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.176+0000 7f9d20c32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:42:11.204 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.202+0000 7fe7ed732640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "49182b31-63d6-416d-a58c-71143c8d99f2", "id": 2}] 2024-03-29T11:42:11.205 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.202+0000 7fe7ed732640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "49182b31-63d6-416d-a58c-71143c8d99f2", "id": 2}] 2024-03-29T11:42:11.205 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.202+0000 7fe7ed732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.205 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.202+0000 7fe7ed732640 1 -- 172.21.15.171:0/2290188523 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "49182b31-63d6-416d-a58c-71143c8d99f2", "id": 2} v 0) v1 -- 0x7fe7c80058b0 con 0x7fe7e80b8240 2024-03-29T11:42:11.215 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.212+0000 7fe7cf7fe640 1 -- 172.21.15.171:0/2290188523 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "49182b31-63d6-416d-a58c-71143c8d99f2", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4076453525 0 216599125) 0x7fe7dc006250 con 0x7fe7e80b8240 2024-03-29T11:42:11.215 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.212+0000 7fe7cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "49182b31-63d6-416d-a58c-71143c8d99f2", "id": 2}] 2024-03-29T11:42:11.215 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.212+0000 7fe7cf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:11.215 INFO:teuthology.orchestra.run.smithi171.stdout:2 2024-03-29T11:42:11.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 10 librados: watch_flush enter 2024-03-29T11:42:11.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 10 librados: watch_flush exit 2024-03-29T11:42:11.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:42:11.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:42:11.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 10 monclient: shutdown 2024-03-29T11:42:11.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 1 -- 172.21.15.171:0/2290188523 >> v1:172.21.15.171:6789/0 conn(0x7fe7e80b8240 legacy=0x7fe7e8134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 1 -- 172.21.15.171:0/2290188523 shutdown_connections 2024-03-29T11:42:11.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 1 -- 172.21.15.171:0/2290188523 >> 172.21.15.171:0/2290188523 conn(0x7fe7e8066b00 msgr2=0x7fe7e806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 1 -- 172.21.15.171:0/2290188523 shutdown_connections 2024-03-29T11:42:11.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 1 -- 172.21.15.171:0/2290188523 wait complete. 2024-03-29T11:42:11.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.213+0000 7fe7ed732640 1 librados: shutdown 2024-03-29T11:42:11.227 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 07800f7f-877c-404b-9c2b-d1de99c44a46 3 2024-03-29T11:42:11.228 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.226+0000 7fa3002c5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:42:11.234 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.232+0000 7f9d20c32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:42:11.275 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.272+0000 7fa3002c5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:42:11.282 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.279+0000 7f9d20c32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:42:11.327 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.324+0000 7f0b8f5cb640 10 monclient: get_monmap_and_config 2024-03-29T11:42:11.327 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.324+0000 7f0b8f5cb640 10 monclient: build_initial_monmap 2024-03-29T11:42:11.327 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.324+0000 7f0b8f5cb640 10 monclient: monmap: 2024-03-29T11:42:11.327 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:11.327 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.327 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:42:11.325623+0000 2024-03-29T11:42:11.327 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:42:11.325623+0000 2024-03-29T11:42:11.328 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:11.328 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.328 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:11.328 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:11.328 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:11.328 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.336 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.328+0000 7fa3002c5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.325+0000 7f0b8f5cb640 1 Processor -- start 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.325+0000 7f0b8f5cb640 1 -- start start 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.325+0000 7f0b8f5cb640 10 monclient: init 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.325+0000 7f0b8f5cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.325+0000 7f0b8f5cb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 10 monclient(hunting): picked mon.noname-b con 0x7f0b8809ac40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 10 monclient(hunting): picked mon.noname-a con 0x7f0b8809c870 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 10 monclient(hunting): picked mon.noname-c con 0x7f0b880b4360 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8809dec0 con 0x7f0b8809c870 2024-03-29T11:42:11.337 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8809da30 con 0x7f0b880b4360 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b880b39c0 con 0x7f0b8809ac40 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 10 monclient(hunting): authenticate will time out at 935.626022s 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8db68640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0b880b4360 0x7f0b8814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42792/0 (socket says 172.21.15.171:42792) 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8db68640 1 -- 172.21.15.171:0/2272056090 learned_addr learned my addr 172.21.15.171:0/2272056090 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553274478 0 0) 0x7f0b8809da30 con 0x7f0b880b4360 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b74003520 con 0x7f0b880b4360 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081859437 0 0) 0x7f0b880b39c0 con 0x7f0b8809ac40 2024-03-29T11:42:11.338 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8809da30 con 0x7f0b8809ac40 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230196865 0 0) 0x7f0b78001110 con 0x7f0b8809c870 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b880b39c0 con 0x7f0b8809c870 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1264470412 0 0) 0x7f0b74003520 con 0x7f0b880b4360 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 >> v1:172.21.15.55:6790/0 conn(0x7f0b8809ac40 legacy=0x7f0b880b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 >> v1:172.21.15.55:6789/0 conn(0x7f0b8809c870 legacy=0x7f0b8813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.339 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 monclient: found mon.noname-c 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b880b5160 con 0x7f0b880b4360 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0b80001a10 con 0x7f0b880b4360 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: dump: 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.340 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b80001bf0 con 0x7f0b880b4360 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 1 -- 172.21.15.171:0/2272056090 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0b80002000 con 0x7f0b880b4360 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b8f5cb640 4 set_mon_vals no callback set 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.341 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.326+0000 7f0b7ffff640 10 monclient: dump: 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.327+0000 7f0b8f5cb640 10 monclient: shutdown 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.327+0000 7f0b8f5cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.327+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/2272056090 >> v1:172.21.15.171:6789/0 conn(0x7f0b880b4360 legacy=0x7f0b8814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.327+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/2272056090 shutdown_connections 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.327+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/2272056090 >> 172.21.15.171:0/2272056090 conn(0x7f0b88066b00 msgr2=0x7f0b88066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.327+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/2272056090 shutdown_connections 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.327+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/2272056090 wait complete. 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 10 monclient: build_initial_monmap 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 10 monclient: monmap: 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:11.343 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 1 librados: starting msgr at 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 1 librados: starting objecter 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 1 Processor -- start 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 1 -- start start 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 1 librados: setting wanted keys 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 1 librados: calling monclient init 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 10 monclient: init 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.328+0000 7f0b8f5cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 10 monclient(hunting): picked mon.noname-c con 0x7f0b8809ac40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 10 monclient(hunting): picked mon.noname-a con 0x7f0b881d8690 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 10 monclient(hunting): picked mon.noname-b con 0x7f0b881455c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:11.344 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b881d7af0 con 0x7f0b881d8690 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b88145a10 con 0x7f0b881455c0 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b880b5a20 con 0x7f0b8809ac40 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 10 monclient(hunting): authenticate will time out at 935.629011s 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8db68640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0b881455c0 0x7f0b881e2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42798/0 (socket says 172.21.15.171:42798) 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8db68640 1 -- 172.21.15.171:0/850415302 learned_addr learned my addr 172.21.15.171:0/850415302 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 client.?.objecter ms_handle_connect 0x7f0b881455c0 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 client.?.objecter ms_handle_connect 0x7f0b8809ac40 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 client.?.objecter ms_handle_connect 0x7f0b881d8690 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515102594 0 0) 0x7f0b88145a10 con 0x7f0b881455c0 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.345 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b6c004eb0 con 0x7f0b881455c0 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702558934 0 0) 0x7f0b880b5a20 con 0x7f0b8809ac40 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b88145a10 con 0x7f0b8809ac40 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089662115 0 0) 0x7f0b881d7af0 con 0x7f0b881d8690 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:42:11.346 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b880b5a20 con 0x7f0b881d8690 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1614405318 0 0) 0x7f0b6c004eb0 con 0x7f0b881455c0 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b881d7af0 con 0x7f0b881455c0 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0b80001f40 con 0x7f0b881455c0 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): dump: 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.347 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2661231325 0 0) 0x7f0b88145a10 con 0x7f0b8809ac40 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b6c004eb0 con 0x7f0b8809ac40 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0b84001930 con 0x7f0b8809ac40 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): dump: 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.348 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3431824338 0 0) 0x7f0b880b5a20 con 0x7f0b881d8690 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b88145a10 con 0x7f0b881d8690 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0b78001860 con 0x7f0b881d8690 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient(hunting): dump: 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.349 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2239212026 0 0) 0x7f0b881d7af0 con 0x7f0b881455c0 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 >> v1:172.21.15.55:6790/0 conn(0x7f0b8809ac40 legacy=0x7f0b881d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 >> v1:172.21.15.55:6789/0 conn(0x7f0b881d8690 legacy=0x7f0b88144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 monclient: found mon.b 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b88141960 con 0x7f0b881455c0 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:11.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 10 monclient: _renew_subs 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b881e3b20 con 0x7f0b881455c0 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b80004e10 con 0x7f0b881455c0 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0b80005120 con 0x7f0b881455c0 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 10 monclient: dump: 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.351 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0b80005610 con 0x7f0b881455c0 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.329+0000 7f0b7f7fe640 4 set_mon_vals no callback set 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b8f5cb640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b8f5cb640 10 monclient: _renew_subs 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b8f5cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b881e3e90 con 0x7f0b881455c0 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b8f5cb640 1 librados: init done 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1760080866 0 0) 0x7f0b80005f30 con 0x7f0b881455c0 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b7dffb640 10 client.4132.objecter ms_dispatch 0x7f0b88000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:42:11.352 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b7dffb640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:42:11.353 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b7dffb640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:42:11.353 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b7dffb640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:42:11.353 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b8f5cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.353 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b8f5cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.353 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b8f5cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.353 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.330+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b88141b50 con 0x7f0b881455c0 2024-03-29T11:42:11.353 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.332+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0b80006170 con 0x7f0b881455c0 2024-03-29T11:42:11.353 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.332+0000 7f0b7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.353 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.332+0000 7f0b7dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:11.353 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.343+0000 7f9d20c32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:42:11.375 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.372+0000 7fa3002c5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:42:11.390 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.387+0000 7f9d20c32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:42:11.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.404+0000 7f0b8f5cb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "07800f7f-877c-404b-9c2b-d1de99c44a46", "id": 3}] 2024-03-29T11:42:11.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.404+0000 7f0b8f5cb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "07800f7f-877c-404b-9c2b-d1de99c44a46", "id": 3}] 2024-03-29T11:42:11.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.404+0000 7f0b8f5cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.404+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/850415302 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "07800f7f-877c-404b-9c2b-d1de99c44a46", "id": 3} v 0) v1 -- 0x7f0b8804f730 con 0x7f0b881455c0 2024-03-29T11:42:11.419 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.417+0000 7f0b7dffb640 1 -- 172.21.15.171:0/850415302 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07800f7f-877c-404b-9c2b-d1de99c44a46", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3627165463 0 525046306) 0x7f0b80005aa0 con 0x7f0b881455c0 2024-03-29T11:42:11.419 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.417+0000 7f0b7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "07800f7f-877c-404b-9c2b-d1de99c44a46", "id": 3}] 2024-03-29T11:42:11.419 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.417+0000 7f0b7dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:11.420 INFO:teuthology.orchestra.run.smithi171.stdout:3 2024-03-29T11:42:11.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 10 librados: watch_flush enter 2024-03-29T11:42:11.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 10 librados: watch_flush exit 2024-03-29T11:42:11.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:42:11.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:42:11.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 10 monclient: shutdown 2024-03-29T11:42:11.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/850415302 >> v1:172.21.15.171:6789/0 conn(0x7f0b881455c0 legacy=0x7f0b881e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/850415302 shutdown_connections 2024-03-29T11:42:11.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/850415302 >> 172.21.15.171:0/850415302 conn(0x7f0b88066b00 msgr2=0x7f0b8813a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/850415302 shutdown_connections 2024-03-29T11:42:11.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 1 -- 172.21.15.171:0/850415302 wait complete. 2024-03-29T11:42:11.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.418+0000 7f0b8f5cb640 1 librados: shutdown 2024-03-29T11:42:11.431 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 69fce8b9-b724-4494-be9c-77fe3b3a509e 4 2024-03-29T11:42:11.495 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.492+0000 7fa3002c5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:42:11.511 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.509+0000 7f9d20c32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:42:11.567 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.564+0000 7fa3002c5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:42:11.571 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.569+0000 7f02f389a640 10 monclient: get_monmap_and_config 2024-03-29T11:42:11.571 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.569+0000 7f02f389a640 10 monclient: build_initial_monmap 2024-03-29T11:42:11.571 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.569+0000 7f02f389a640 10 monclient: monmap: 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:42:11.570198+0000 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:42:11.570198+0000 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:11.572 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.569+0000 7f02f389a640 1 Processor -- start 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 1 -- start start 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 10 monclient: init 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 10 monclient(hunting): picked mon.noname-c con 0x7f02ec0af360 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 10 monclient(hunting): picked mon.noname-b con 0x7f02ec06af60 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 10 monclient(hunting): picked mon.noname-a con 0x7f02ec145e60 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec0b0520 con 0x7f02ec145e60 2024-03-29T11:42:11.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec0aff20 con 0x7f02ec0af360 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec0b1da0 con 0x7f02ec06af60 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f389a640 10 monclient(hunting): authenticate will time out at 935.870636s 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f1636640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f02ec0af360 0x7f02ec06c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42810/0 (socket says 172.21.15.171:42810) 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.570+0000 7f02f1636640 1 -- 172.21.15.171:0/409318370 learned_addr learned my addr 172.21.15.171:0/409318370 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098735205 0 0) 0x7f02ec0aff20 con 0x7f02ec0af360 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d8003520 con 0x7f02ec0af360 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358340402 0 0) 0x7f02ec0b1da0 con 0x7f02ec06af60 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ec0aff20 con 0x7f02ec06af60 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584014380 0 0) 0x7f02ec0b0520 con 0x7f02ec145e60 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ec0b1da0 con 0x7f02ec145e60 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749492866 0 0) 0x7f02d8003520 con 0x7f02ec0af360 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 >> v1:172.21.15.55:6790/0 conn(0x7f02ec06af60 legacy=0x7f02ec145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 >> v1:172.21.15.55:6789/0 conn(0x7f02ec145e60 legacy=0x7f02ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 monclient: found mon.noname-c 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:11.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec0b2080 con 0x7f02ec0af360 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02dc001690 con 0x7f02ec0af360 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02f389a640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02f389a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: dump: 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02f389a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02dc001870 con 0x7f02ec0af360 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02f389a640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 1 -- 172.21.15.171:0/409318370 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02dc001c00 con 0x7f02ec0af360 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02f389a640 4 set_mon_vals no callback set 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02e3fff640 10 monclient: dump: 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02f389a640 10 monclient: shutdown 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02f389a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.571+0000 7f02f389a640 1 -- 172.21.15.171:0/409318370 >> v1:172.21.15.171:6789/0 conn(0x7f02ec0af360 legacy=0x7f02ec06c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.572+0000 7f02f389a640 1 -- 172.21.15.171:0/409318370 shutdown_connections 2024-03-29T11:42:11.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.572+0000 7f02f389a640 1 -- 172.21.15.171:0/409318370 >> 172.21.15.171:0/409318370 conn(0x7f02ec0b58f0 msgr2=0x7f02ec0b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.572+0000 7f02f389a640 1 -- 172.21.15.171:0/409318370 shutdown_connections 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.572+0000 7f02f389a640 1 -- 172.21.15.171:0/409318370 wait complete. 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.572+0000 7f02f389a640 10 monclient: build_initial_monmap 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.572+0000 7f02f389a640 10 monclient: monmap: 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.572+0000 7f02f389a640 1 librados: starting msgr at 2024-03-29T11:42:11.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.572+0000 7f02f389a640 1 librados: starting objecter 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 1 Processor -- start 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 1 -- start start 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 1 librados: setting wanted keys 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 1 librados: calling monclient init 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 10 monclient: init 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 10 monclient(hunting): picked mon.noname-c con 0x7f02ec06af60 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 10 monclient(hunting): picked mon.noname-a con 0x7f02ec0af360 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 10 monclient(hunting): picked mon.noname-b con 0x7f02ec145e60 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec1d7f20 con 0x7f02ec0af360 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec1d7a00 con 0x7f02ec145e60 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec13d760 con 0x7f02ec06af60 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:11.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f389a640 10 monclient(hunting): authenticate will time out at 935.873580s 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f1e37640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f02ec145e60 0x7f02ec1e28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42824/0 (socket says 172.21.15.171:42824) 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02f1e37640 1 -- 172.21.15.171:0/2451175741 learned_addr learned my addr 172.21.15.171:0/2451175741 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 client.?.objecter ms_handle_connect 0x7f02ec145e60 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 client.?.objecter ms_handle_connect 0x7f02ec06af60 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 client.?.objecter ms_handle_connect 0x7f02ec0af360 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210055622 0 0) 0x7f02ec1d7a00 con 0x7f02ec145e60 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.573+0000 7f02e1ffb640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:42:11.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d0004eb0 con 0x7f02ec145e60 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213454120 0 0) 0x7f02ec13d760 con 0x7f02ec06af60 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ec1d7a00 con 0x7f02ec06af60 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908600909 0 0) 0x7f02ec1d7f20 con 0x7f02ec0af360 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ec13d760 con 0x7f02ec0af360 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897159848 0 0) 0x7f02ec1d7a00 con 0x7f02ec06af60 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02ec1d7f20 con 0x7f02ec06af60 2024-03-29T11:42:11.591 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02dc001b80 con 0x7f02ec06af60 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1536032990 0 0) 0x7f02d0004eb0 con 0x7f02ec145e60 2024-03-29T11:42:11.592 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02ec1d7a00 con 0x7f02ec145e60 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 157786200 0 0) 0x7f02ec13d760 con 0x7f02ec0af360 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02d0004eb0 con 0x7f02ec0af360 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02e4004250 con 0x7f02ec145e60 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.593 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02e80018b0 con 0x7f02ec0af360 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.594 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2208150966 0 0) 0x7f02ec1d7f20 con 0x7f02ec06af60 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 >> v1:172.21.15.171:6789/0 conn(0x7f02ec145e60 legacy=0x7f02ec1e28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 >> v1:172.21.15.55:6789/0 conn(0x7f02ec0af360 legacy=0x7f02ec13da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 monclient: found mon.c 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec13eae0 con 0x7f02ec06af60 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02f389a640 5 monclient: authenticate success, global_id 4136 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02f389a640 10 monclient: _renew_subs 2024-03-29T11:42:11.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02f389a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02f389a640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ec13ed90 con 0x7f02ec06af60 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02dc004a90 con 0x7f02ec06af60 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02dc004c40 con 0x7f02ec06af60 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 monclient: dump: 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.596 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e37fe640 4 set_mon_vals no callback set 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02f389a640 10 client.4136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02f389a640 10 monclient: _renew_subs 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f02dc005130 con 0x7f02ec06af60 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02f389a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02f389a640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ec1d7670 con 0x7f02ec06af60 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02f389a640 1 librados: init done 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3933931616 0 0) 0x7f02dc0011e0 con 0x7f02ec06af60 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 10 client.4136.objecter ms_dispatch 0x7f02ec000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 3 client.4136.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 3 client.4136.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.574+0000 7f02e1ffb640 20 client.4136.objecter dump_active .. 0 homeless 2024-03-29T11:42:11.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.575+0000 7f02f389a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.575+0000 7f02f389a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.575+0000 7f02f389a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:11.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.575+0000 7f02f389a640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ec067c00 con 0x7f02ec06af60 2024-03-29T11:42:11.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.577+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f02dc005490 con 0x7f02ec06af60 2024-03-29T11:42:11.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.577+0000 7f02e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.577+0000 7f02e1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:11.598 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.583+0000 7f9d20c32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:42:11.615 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.613+0000 7fa3002c5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:42:11.634 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.631+0000 7f9d20c32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:42:11.652 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.649+0000 7f02f389a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "69fce8b9-b724-4494-be9c-77fe3b3a509e", "id": 4}] 2024-03-29T11:42:11.652 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.649+0000 7f02f389a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "69fce8b9-b724-4494-be9c-77fe3b3a509e", "id": 4}] 2024-03-29T11:42:11.652 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.649+0000 7f02f389a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:11.652 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.649+0000 7f02f389a640 1 -- 172.21.15.171:0/2451175741 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd new", "uuid": "69fce8b9-b724-4494-be9c-77fe3b3a509e", "id": 4} v 0) v1 -- 0x7f02ec1d7c40 con 0x7f02ec06af60 2024-03-29T11:42:11.664 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.662+0000 7f02e1ffb640 1 -- 172.21.15.171:0/2451175741 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69fce8b9-b724-4494-be9c-77fe3b3a509e", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3181834149 0 1697012583) 0x7f02ec1d7c40 con 0x7f02ec06af60 2024-03-29T11:42:11.664 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.662+0000 7f02e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "69fce8b9-b724-4494-be9c-77fe3b3a509e", "id": 4}] 2024-03-29T11:42:11.664 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.662+0000 7f02e1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:11.665 INFO:teuthology.orchestra.run.smithi171.stdout:4 2024-03-29T11:42:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.662+0000 7f02f389a640 10 librados: watch_flush enter 2024-03-29T11:42:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.662+0000 7f02f389a640 10 librados: watch_flush exit 2024-03-29T11:42:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.662+0000 7f02f389a640 20 client.4136.objecter shutdown clearing up homeless session... 2024-03-29T11:42:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.662+0000 7f02f389a640 10 client.4136.objecter successfully canceled tick 2024-03-29T11:42:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.663+0000 7f02f389a640 10 monclient: shutdown 2024-03-29T11:42:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.663+0000 7f02f389a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.663+0000 7f02f389a640 1 -- 172.21.15.171:0/2451175741 >> v1:172.21.15.55:6790/0 conn(0x7f02ec06af60 legacy=0x7f02ec13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.663+0000 7f02f389a640 1 -- 172.21.15.171:0/2451175741 shutdown_connections 2024-03-29T11:42:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.663+0000 7f02f389a640 1 -- 172.21.15.171:0/2451175741 >> 172.21.15.171:0/2451175741 conn(0x7f02ec0b58f0 msgr2=0x7f02ec0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.667 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.663+0000 7f02f389a640 1 -- 172.21.15.171:0/2451175741 shutdown_connections 2024-03-29T11:42:11.667 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.663+0000 7f02f389a640 1 -- 172.21.15.171:0/2451175741 wait complete. 2024-03-29T11:42:11.667 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.663+0000 7f02f389a640 1 librados: shutdown 2024-03-29T11:42:11.670 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.667+0000 7fa3002c5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:42:11.675 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 3c44f523-b8c3-480d-8239-3170a17ba3e2 5 2024-03-29T11:42:11.689 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.686+0000 7f9d20c32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:42:11.719 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:11.716+0000 7fa3002c5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:42:11.739 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:11.736+0000 7f9d20c32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:42:11.789 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.786+0000 7f6483e25640 10 monclient: get_monmap_and_config 2024-03-29T11:42:11.790 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.786+0000 7f6483e25640 10 monclient: build_initial_monmap 2024-03-29T11:42:11.790 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.786+0000 7f6483e25640 10 monclient: monmap: 2024-03-29T11:42:11.790 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:11.790 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.790 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:42:11.787277+0000 2024-03-29T11:42:11.791 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:42:11.787277+0000 2024-03-29T11:42:11.791 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:11.791 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.791 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:11.791 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:11.791 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:11.792 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.786+0000 7f6483e25640 1 Processor -- start 2024-03-29T11:42:11.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 1 -- start start 2024-03-29T11:42:11.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 10 monclient: init 2024-03-29T11:42:11.804 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:11.804 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 10 monclient(hunting): picked mon.noname-c con 0x7f647c09ac40 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 10 monclient(hunting): picked mon.noname-b con 0x7f647c09c870 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 10 monclient(hunting): picked mon.noname-a con 0x7f647c0b4360 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c09dec0 con 0x7f647c0b4360 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c09da30 con 0x7f647c09ac40 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c0b39c0 con 0x7f647c09c870 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6483e25640 10 monclient(hunting): authenticate will time out at 936.087600s 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6481bc1640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f647c09ac40 0x7f647c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42840/0 (socket says 172.21.15.171:42840) 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.787+0000 7f6481bc1640 1 -- 172.21.15.171:0/1404892791 learned_addr learned my addr 172.21.15.171:0/1404892791 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575779188 0 0) 0x7f647c09da30 con 0x7f647c09ac40 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.805 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:42:11.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6468003520 con 0x7f647c09ac40 2024-03-29T11:42:11.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866615863 0 0) 0x7f647c0b39c0 con 0x7f647c09c870 2024-03-29T11:42:11.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c09da30 con 0x7f647c09c870 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142467010 0 0) 0x7f6474001490 con 0x7f647c0b4360 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c0b39c0 con 0x7f647c0b4360 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 663398310 0 0) 0x7f6468003520 con 0x7f647c09ac40 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 >> v1:172.21.15.55:6790/0 conn(0x7f647c09c870 legacy=0x7f647c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 >> v1:172.21.15.55:6789/0 conn(0x7f647c0b4360 legacy=0x7f647c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: _finish_hunting 0 2024-03-29T11:42:11.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 monclient: found mon.noname-c 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c0b5160 con 0x7f647c09ac40 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: _finish_auth 0 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: _check_auth_tickets 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6478001690 con 0x7f647c09ac40 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: dump: 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.808 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6483e25640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6483e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6478001870 con 0x7f647c09ac40 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 1 -- 172.21.15.171:0/1404892791 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6478001c80 con 0x7f647c09ac40 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6483e25640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:11.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6483e25640 4 set_mon_vals no callback set 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6480bbf640 10 monclient: dump: 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.810 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6483e25640 10 monclient: shutdown 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6483e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.788+0000 7f6483e25640 1 -- 172.21.15.171:0/1404892791 >> v1:172.21.15.171:6789/0 conn(0x7f647c09ac40 legacy=0x7f647c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.789+0000 7f6483e25640 1 -- 172.21.15.171:0/1404892791 shutdown_connections 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.789+0000 7f6483e25640 1 -- 172.21.15.171:0/1404892791 >> 172.21.15.171:0/1404892791 conn(0x7f647c066b00 msgr2=0x7f647c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.789+0000 7f6483e25640 1 -- 172.21.15.171:0/1404892791 shutdown_connections 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.789+0000 7f6483e25640 1 -- 172.21.15.171:0/1404892791 wait complete. 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.789+0000 7f6483e25640 10 monclient: build_initial_monmap 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.789+0000 7f6483e25640 10 monclient: monmap: 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:11.811 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.789+0000 7f6483e25640 1 librados: starting msgr at 2024-03-29T11:42:11.812 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.789+0000 7f6483e25640 1 librados: starting objecter 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 1 Processor -- start 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 1 -- start start 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 1 librados: setting wanted keys 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 1 librados: calling monclient init 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 10 monclient: init 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 10 monclient(hunting): picked mon.noname-a con 0x7f647c09ac40 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 10 monclient(hunting): picked mon.noname-c con 0x7f647c0af1d0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 10 monclient(hunting): picked mon.noname-b con 0x7f647c139370 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c1d4be0 con 0x7f647c09ac40 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c0b06a0 con 0x7f647c139370 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c0b1910 con 0x7f647c0af1d0 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:11.813 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f6483e25640 10 monclient(hunting): authenticate will time out at 936.090551s 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64823c2640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f647c139370 0x7f647c1d70e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42854/0 (socket says 172.21.15.171:42854) 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64823c2640 1 -- 172.21.15.171:0/3851876278 learned_addr learned my addr 172.21.15.171:0/3851876278 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 client.?.objecter ms_handle_connect 0x7f647c139370 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 client.?.objecter ms_handle_connect 0x7f647c09ac40 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 client.?.objecter ms_handle_connect 0x7f647c0af1d0 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376529440 0 0) 0x7f647c0b06a0 con 0x7f647c139370 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:42:11.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6460004eb0 con 0x7f647c139370 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392995415 0 0) 0x7f647c1d4be0 con 0x7f647c09ac40 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.790+0000 7f64727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c0b06a0 con 0x7f647c09ac40 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195663075 0 0) 0x7f647c0b1910 con 0x7f647c0af1d0 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c1d4be0 con 0x7f647c0af1d0 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2452338351 0 0) 0x7f6460004eb0 con 0x7f647c139370 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f647c0b1910 con 0x7f647c139370 2024-03-29T11:42:11.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6474001a40 con 0x7f647c139370 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): dump: 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.816 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4249193286 0 0) 0x7f647c0b06a0 con 0x7f647c09ac40 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6460004eb0 con 0x7f647c09ac40 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6478001b80 con 0x7f647c09ac40 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient(hunting): dump: 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.817 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3583400296 0 0) 0x7f647c0b1910 con 0x7f647c139370 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 >> v1:172.21.15.55:6790/0 conn(0x7f647c0af1d0 legacy=0x7f647c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 >> v1:172.21.15.55:6789/0 conn(0x7f647c09ac40 legacy=0x7f647c0afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 monclient: found mon.b 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c139970 con 0x7f647c139370 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6483e25640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6483e25640 10 monclient: _renew_subs 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6483e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6483e25640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c1d77f0 con 0x7f647c139370 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6483e25640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6483e25640 10 monclient: _renew_subs 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6483e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6483e25640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c1d7d70 con 0x7f647c139370 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6483e25640 1 librados: init done 2024-03-29T11:42:11.819 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6474001f30 con 0x7f647c139370 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6474003b30 con 0x7f647c139370 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 monclient: dump: 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:11.820 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f6473fff640 4 set_mon_vals no callback set 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f64740040e0 con 0x7f647c139370 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3309497908 0 0) 0x7f64740068d0 con 0x7f647c139370 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 10 client.4144.objecter ms_dispatch 0x7f647c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.791+0000 7f64727fc640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.792+0000 7f6483e25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.792+0000 7f6483e25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.792+0000 7f6483e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.821 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.792+0000 7f6483e25640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6464005080 con 0x7f647c139370 2024-03-29T11:42:11.822 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.794+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6474004520 con 0x7f647c139370 2024-03-29T11:42:11.822 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.794+0000 7f64727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:11.822 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.794+0000 7f64727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:11.887 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.885+0000 7f6483e25640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3c44f523-b8c3-480d-8239-3170a17ba3e2", "id": 5}] 2024-03-29T11:42:11.888 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.885+0000 7f6483e25640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3c44f523-b8c3-480d-8239-3170a17ba3e2", "id": 5}] 2024-03-29T11:42:11.888 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.885+0000 7f6483e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:11.888 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.885+0000 7f6483e25640 1 -- 172.21.15.171:0/3851876278 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3c44f523-b8c3-480d-8239-3170a17ba3e2", "id": 5} v 0) v1 -- 0x7f64640058b0 con 0x7f647c139370 2024-03-29T11:42:11.897 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f64727fc640 1 -- 172.21.15.171:0/3851876278 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3c44f523-b8c3-480d-8239-3170a17ba3e2", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3015370835 0 1988412176) 0x7f6474004780 con 0x7f647c139370 2024-03-29T11:42:11.897 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f64727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3c44f523-b8c3-480d-8239-3170a17ba3e2", "id": 5}] 2024-03-29T11:42:11.897 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f64727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:11.898 INFO:teuthology.orchestra.run.smithi171.stdout:5 2024-03-29T11:42:11.900 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f6483e25640 10 librados: watch_flush enter 2024-03-29T11:42:11.900 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f6483e25640 10 librados: watch_flush exit 2024-03-29T11:42:11.900 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f6483e25640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:42:11.900 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f6483e25640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:42:11.900 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f6483e25640 10 monclient: shutdown 2024-03-29T11:42:11.900 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f6483e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.901 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f6483e25640 1 -- 172.21.15.171:0/3851876278 >> v1:172.21.15.171:6789/0 conn(0x7f647c139370 legacy=0x7f647c1d70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.901 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f6483e25640 1 -- 172.21.15.171:0/3851876278 shutdown_connections 2024-03-29T11:42:11.901 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.895+0000 7f6483e25640 1 -- 172.21.15.171:0/3851876278 >> 172.21.15.171:0/3851876278 conn(0x7f647c066b00 msgr2=0x7f647c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.901 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.896+0000 7f6483e25640 1 -- 172.21.15.171:0/3851876278 shutdown_connections 2024-03-29T11:42:11.901 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.896+0000 7f6483e25640 1 -- 172.21.15.171:0/3851876278 wait complete. 2024-03-29T11:42:11.901 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:11.896+0000 7f6483e25640 1 librados: shutdown 2024-03-29T11:42:11.908 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 546bd470-ae95-4400-b122-5629405751e6 6 2024-03-29T11:42:12.009 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.007+0000 7f87039ab640 10 monclient: get_monmap_and_config 2024-03-29T11:42:12.010 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.007+0000 7f87039ab640 10 monclient: build_initial_monmap 2024-03-29T11:42:12.010 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.007+0000 7f87039ab640 10 monclient: monmap: 2024-03-29T11:42:12.010 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:12.010 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.010 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:42:12.008006+0000 2024-03-29T11:42:12.010 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:42:12.008006+0000 2024-03-29T11:42:12.011 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:12.011 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.011 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:12.011 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:12.011 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:12.011 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.007+0000 7f87039ab640 1 Processor -- start 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.007+0000 7f87039ab640 1 -- start start 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.007+0000 7f87039ab640 10 monclient: init 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 10 monclient(hunting): picked mon.noname-b con 0x7f86fc09ac40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 10 monclient(hunting): picked mon.noname-c con 0x7f86fc09c870 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 10 monclient(hunting): picked mon.noname-a con 0x7f86fc0b4360 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc09dec0 con 0x7f86fc0b4360 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc09da30 con 0x7f86fc09c870 2024-03-29T11:42:12.022 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc0b39c0 con 0x7f86fc09ac40 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f87039ab640 10 monclient(hunting): authenticate will time out at 936.308505s 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f8700f46640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f86fc09c870 0x7f86fc13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42858/0 (socket says 172.21.15.171:42858) 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f8700f46640 1 -- 172.21.15.171:0/2052972276 learned_addr learned my addr 172.21.15.171:0/2052972276 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683674017 0 0) 0x7f86fc09da30 con 0x7f86fc09c870 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f86f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f86f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f86f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f86f3fff640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e8003520 con 0x7f86fc09c870 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.008+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835603053 0 0) 0x7f86f4001490 con 0x7f86fc0b4360 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:42:12.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc09da30 con 0x7f86fc0b4360 2024-03-29T11:42:12.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086968804 0 0) 0x7f86fc0b39c0 con 0x7f86fc09ac40 2024-03-29T11:42:12.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f4001490 con 0x7f86fc09ac40 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1833446194 0 0) 0x7f86e8003520 con 0x7f86fc09c870 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 >> v1:172.21.15.55:6790/0 conn(0x7f86fc09ac40 legacy=0x7f86fc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 >> v1:172.21.15.55:6789/0 conn(0x7f86fc0b4360 legacy=0x7f86fc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 monclient: found mon.noname-c 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc0b5160 con 0x7f86fc09c870 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:12.025 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f87039ab640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f86f8001690 con 0x7f86fc09c870 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f87039ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: dump: 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.026 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f87039ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86f8001870 con 0x7f86fc09c870 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f87039ab640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:12.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f87039ab640 4 set_mon_vals no callback set 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 1 -- 172.21.15.171:0/2052972276 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f86f8001c80 con 0x7f86fc09c870 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.009+0000 7f86f3fff640 10 monclient: dump: 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.028 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.010+0000 7f87039ab640 10 monclient: shutdown 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.010+0000 7f87039ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.010+0000 7f87039ab640 1 -- 172.21.15.171:0/2052972276 >> v1:172.21.15.171:6789/0 conn(0x7f86fc09c870 legacy=0x7f86fc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.010+0000 7f87039ab640 1 -- 172.21.15.171:0/2052972276 shutdown_connections 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.010+0000 7f87039ab640 1 -- 172.21.15.171:0/2052972276 >> 172.21.15.171:0/2052972276 conn(0x7f86fc066b00 msgr2=0x7f86fc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.010+0000 7f87039ab640 1 -- 172.21.15.171:0/2052972276 shutdown_connections 2024-03-29T11:42:12.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.010+0000 7f87039ab640 1 -- 172.21.15.171:0/2052972276 wait complete. 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.010+0000 7f87039ab640 10 monclient: build_initial_monmap 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.010+0000 7f87039ab640 10 monclient: monmap: 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:12.030 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 1 librados: starting msgr at 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 1 librados: starting objecter 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 1 Processor -- start 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 1 -- start start 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 1 librados: setting wanted keys 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 1 librados: calling monclient init 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 10 monclient: init 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 10 monclient(hunting): picked mon.noname-a con 0x7f86fc09ac40 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 10 monclient(hunting): picked mon.noname-b con 0x7f86fc0b6b00 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:12.031 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 10 monclient(hunting): picked mon.noname-c con 0x7f86fc1406b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc1d7e20 con 0x7f86fc09ac40 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc140b00 con 0x7f86fc0b6b00 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc0ad080 con 0x7f86fc1406b0 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f87039ab640 10 monclient(hunting): authenticate will time out at 936.311660s 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f8700f46640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f86fc0b6b00 0x7f86fc13ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42868/0 (socket says 172.21.15.171:42868) 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f8700f46640 1 -- 172.21.15.171:0/1712907505 learned_addr learned my addr 172.21.15.171:0/1712907505 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f86f1ffb640 10 client.?.objecter ms_handle_connect 0x7f86fc0b6b00 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f86f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f86f1ffb640 10 client.?.objecter ms_handle_connect 0x7f86fc09ac40 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.011+0000 7f86f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.032 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 client.?.objecter ms_handle_connect 0x7f86fc1406b0 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024668007 0 0) 0x7f86fc1d7e20 con 0x7f86fc09ac40 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e0004eb0 con 0x7f86fc09ac40 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332370183 0 0) 0x7f86fc140b00 con 0x7f86fc0b6b00 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc1d7e20 con 0x7f86fc0b6b00 2024-03-29T11:42:12.033 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889514888 0 0) 0x7f86fc0ad080 con 0x7f86fc1406b0 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc140b00 con 0x7f86fc1406b0 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3058578339 0 0) 0x7f86e0004eb0 con 0x7f86fc09ac40 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86fc0ad080 con 0x7f86fc09ac40 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f86ec0018f0 con 0x7f86fc09ac40 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.035 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.035 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.035 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.035 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.035 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.035 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.035 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871522403 0 0) 0x7f86fc1d7e20 con 0x7f86fc0b6b00 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86e0004eb0 con 0x7f86fc0b6b00 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f86f8001b80 con 0x7f86fc0b6b00 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.036 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 390787389 0 0) 0x7f86fc140b00 con 0x7f86fc1406b0 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86fc1d7e20 con 0x7f86fc1406b0 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f86f4001a00 con 0x7f86fc1406b0 2024-03-29T11:42:12.037 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.038 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.039 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 503334320 0 0) 0x7f86fc0ad080 con 0x7f86fc09ac40 2024-03-29T11:42:12.039 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 >> v1:172.21.15.55:6790/0 conn(0x7f86fc1406b0 legacy=0x7f86fc1e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.039 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 >> v1:172.21.15.171:6789/0 conn(0x7f86fc0b6b00 legacy=0x7f86fc13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.039 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:12.039 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 monclient: found mon.a 2024-03-29T11:42:12.039 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:12.039 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:12.039 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc1d7ab0 con 0x7f86fc09ac40 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f87039ab640 5 monclient: authenticate success, global_id 4152 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f87039ab640 10 monclient: _renew_subs 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f87039ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f87039ab640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86fc1e3570 con 0x7f86fc09ac40 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86ec003610 con 0x7f86fc09ac40 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:12.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f86ec003920 con 0x7f86fc09ac40 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 10 monclient: dump: 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.041 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.042 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.042 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f37fe640 4 set_mon_vals no callback set 2024-03-29T11:42:12.042 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f87039ab640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:12.042 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f87039ab640 10 monclient: _renew_subs 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f86ec003e10 con 0x7f86fc09ac40 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f87039ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f87039ab640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86fc1e38e0 con 0x7f86fc09ac40 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.012+0000 7f87039ab640 1 librados: init done 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.013+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4159286495 0 0) 0x7f86ec004930 con 0x7f86fc09ac40 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.013+0000 7f86f1ffb640 10 client.4152.objecter ms_dispatch 0x7f86fc000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.013+0000 7f86f1ffb640 3 client.4152.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.013+0000 7f86f1ffb640 3 client.4152.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.013+0000 7f86f1ffb640 20 client.4152.objecter dump_active .. 0 homeless 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.013+0000 7f87039ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.013+0000 7f87039ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.013+0000 7f87039ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.013+0000 7f87039ab640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86fc09b020 con 0x7f86fc09ac40 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.015+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f86ec004340 con 0x7f86fc09ac40 2024-03-29T11:42:12.043 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.015+0000 7f86f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.044 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.015+0000 7f86f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:12.092 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.089+0000 7f87039ab640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "546bd470-ae95-4400-b122-5629405751e6", "id": 6}] 2024-03-29T11:42:12.092 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.089+0000 7f87039ab640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "546bd470-ae95-4400-b122-5629405751e6", "id": 6}] 2024-03-29T11:42:12.092 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.089+0000 7f87039ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:12.092 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.089+0000 7f87039ab640 1 -- 172.21.15.171:0/1712907505 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd new", "uuid": "546bd470-ae95-4400-b122-5629405751e6", "id": 6} v 0) v1 -- 0x7f86fc04f730 con 0x7f86fc09ac40 2024-03-29T11:42:12.093 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.091+0000 7f86f1ffb640 1 -- 172.21.15.171:0/1712907505 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "546bd470-ae95-4400-b122-5629405751e6", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2777812206 0 1113811849) 0x7f86ec004540 con 0x7f86fc09ac40 2024-03-29T11:42:12.093 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.091+0000 7f86f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "546bd470-ae95-4400-b122-5629405751e6", "id": 6}] 2024-03-29T11:42:12.094 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.091+0000 7f86f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:12.094 INFO:teuthology.orchestra.run.smithi171.stdout:6 2024-03-29T11:42:12.095 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 10 librados: watch_flush enter 2024-03-29T11:42:12.095 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 10 librados: watch_flush exit 2024-03-29T11:42:12.095 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 20 client.4152.objecter shutdown clearing up homeless session... 2024-03-29T11:42:12.095 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 10 client.4152.objecter successfully canceled tick 2024-03-29T11:42:12.095 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 10 monclient: shutdown 2024-03-29T11:42:12.095 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:12.095 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 1 -- 172.21.15.171:0/1712907505 >> v1:172.21.15.55:6789/0 conn(0x7f86fc09ac40 legacy=0x7f86fc1d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.096 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 1 -- 172.21.15.171:0/1712907505 shutdown_connections 2024-03-29T11:42:12.096 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 1 -- 172.21.15.171:0/1712907505 >> 172.21.15.171:0/1712907505 conn(0x7f86fc066b00 msgr2=0x7f86fc0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:12.096 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 1 -- 172.21.15.171:0/1712907505 shutdown_connections 2024-03-29T11:42:12.096 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 1 -- 172.21.15.171:0/1712907505 wait complete. 2024-03-29T11:42:12.096 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.092+0000 7f87039ab640 1 librados: shutdown 2024-03-29T11:42:12.105 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new c466d742-1d0d-4f6f-83c1-9572380d0809 7 2024-03-29T11:42:12.116 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:12.114+0000 7fa3002c5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:42:12.160 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:12.157+0000 7f9d20c32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:42:12.205 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.203+0000 7f3aa5cfb640 10 monclient: get_monmap_and_config 2024-03-29T11:42:12.205 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.203+0000 7f3aa5cfb640 10 monclient: build_initial_monmap 2024-03-29T11:42:12.205 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.203+0000 7f3aa5cfb640 10 monclient: monmap: 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:42:12.204017+0000 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:42:12.204017+0000 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:12.206 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.215 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 1 Processor -- start 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 1 -- start start 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 10 monclient: init 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f3aa0066b90 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f3aa013f340 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f3aa013f7a0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa013fc30 con 0x7f3aa0066b90 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa00afd70 con 0x7f3aa013f340 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa00aec00 con 0x7f3aa013f7a0 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3aa5cfb640 10 monclient(hunting): authenticate will time out at 936.504638s 2024-03-29T11:42:12.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3a97fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3aa013f340 0x7f3aa0142060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42874/0 (socket says 172.21.15.171:42874) 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.204+0000 7f3a97fff640 1 -- 172.21.15.171:0/3414755453 learned_addr learned my addr 172.21.15.171:0/3414755453 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173988351 0 0) 0x7f3aa00afd70 con 0x7f3aa013f340 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a8c003520 con 0x7f3aa013f340 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389615696 0 0) 0x7f3aa013fc30 con 0x7f3aa0066b90 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:42:12.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa00afd70 con 0x7f3aa0066b90 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679245805 0 0) 0x7f3aa00aec00 con 0x7f3aa013f7a0 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa013fc30 con 0x7f3aa013f7a0 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 312336368 0 0) 0x7f3a8c003520 con 0x7f3aa013f340 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 >> v1:172.21.15.55:6790/0 conn(0x7f3aa013f7a0 legacy=0x7f3aa0142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 >> v1:172.21.15.55:6789/0 conn(0x7f3aa0066b90 legacy=0x7f3aa00af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 monclient: found mon.noname-c 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa00aed90 con 0x7f3aa013f340 2024-03-29T11:42:12.218 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: _finish_auth 0 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3a88001690 con 0x7f3aa013f340 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3aa5cfb640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3aa5cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: dump: 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.219 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3aa5cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a88001870 con 0x7f3aa013f340 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 1 -- 172.21.15.171:0/3414755453 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3a88001c80 con 0x7f3aa013f340 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3aa5cfb640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3aa5cfb640 4 set_mon_vals no callback set 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.205+0000 7f3a9effd640 10 monclient: dump: 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.206+0000 7f3aa5cfb640 10 monclient: shutdown 2024-03-29T11:42:12.221 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.206+0000 7f3aa5cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.206+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3414755453 >> v1:172.21.15.171:6789/0 conn(0x7f3aa013f340 legacy=0x7f3aa0142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.206+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3414755453 shutdown_connections 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.206+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3414755453 >> 172.21.15.171:0/3414755453 conn(0x7f3aa00b4970 msgr2=0x7f3aa00b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.206+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3414755453 shutdown_connections 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.206+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3414755453 wait complete. 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.206+0000 7f3aa5cfb640 10 monclient: build_initial_monmap 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.206+0000 7f3aa5cfb640 10 monclient: monmap: 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:12.222 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 1 librados: starting msgr at 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 1 librados: starting objecter 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 1 Processor -- start 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 1 -- start start 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 1 librados: setting wanted keys 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 1 librados: calling monclient init 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 10 monclient: init 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:12.223 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f3aa0066b90 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f3aa013f340 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f3aa013f7a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa0130ef0 con 0x7f3aa0066b90 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa0130c60 con 0x7f3aa013f7a0 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa0131510 con 0x7f3aa013f340 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3aa5cfb640 10 monclient(hunting): authenticate will time out at 936.507700s 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3a9ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3aa013f7a0 0x7f3aa01d7b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42886/0 (socket says 172.21.15.171:42886) 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3a9ffff640 1 -- 172.21.15.171:0/3656814900 learned_addr learned my addr 172.21.15.171:0/3656814900 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3a9cff9640 10 client.?.objecter ms_handle_connect 0x7f3aa013f7a0 2024-03-29T11:42:12.224 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3a9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3a9cff9640 10 client.?.objecter ms_handle_connect 0x7f3aa0066b90 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3a9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3a9cff9640 10 client.?.objecter ms_handle_connect 0x7f3aa013f340 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.207+0000 7f3a9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411903407 0 0) 0x7f3aa0130c60 con 0x7f3aa013f7a0 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a80004eb0 con 0x7f3aa013f7a0 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839834232 0 0) 0x7f3aa0130ef0 con 0x7f3aa0066b90 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.225 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa0130c60 con 0x7f3aa0066b90 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036084086 0 0) 0x7f3aa0131510 con 0x7f3aa013f340 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa0130ef0 con 0x7f3aa013f340 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 211681257 0 0) 0x7f3a80004eb0 con 0x7f3aa013f7a0 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3aa0131510 con 0x7f3aa013f7a0 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3a98001c80 con 0x7f3aa013f7a0 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:12.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): dump: 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073440745 0 0) 0x7f3aa0130c60 con 0x7f3aa0066b90 2024-03-29T11:42:12.227 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a80004eb0 con 0x7f3aa0066b90 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3a90001910 con 0x7f3aa0066b90 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): dump: 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3671028053 0 0) 0x7f3aa0130ef0 con 0x7f3aa013f340 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3aa0130c60 con 0x7f3aa013f340 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3a880019e0 con 0x7f3aa013f340 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient(hunting): dump: 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.229 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1260879044 0 0) 0x7f3aa0131510 con 0x7f3aa013f7a0 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 >> v1:172.21.15.55:6790/0 conn(0x7f3aa013f340 legacy=0x7f3aa01d7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 >> v1:172.21.15.55:6789/0 conn(0x7f3aa0066b90 legacy=0x7f3aa01d4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 monclient: found mon.b 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa01316a0 con 0x7f3aa013f7a0 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient: _finish_auth 0 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3aa5cfb640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:42:12.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3aa5cfb640 10 monclient: _renew_subs 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3aa5cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa01d3ce0 con 0x7f3aa013f7a0 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3aa5cfb640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3aa5cfb640 10 monclient: _renew_subs 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3aa5cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa0142d30 con 0x7f3aa013f7a0 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3aa5cfb640 1 librados: init done 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a98003990 con 0x7f3aa013f7a0 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3a98003d60 con 0x7f3aa013f7a0 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9e7fc640 4 set_mon_vals no callback set 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:12.231 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 monclient: dump: 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3a980042d0 con 0x7f3aa013f7a0 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3071558814 0 0) 0x7f3a98006be0 con 0x7f3aa013f7a0 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 10 client.4156.objecter ms_dispatch 0x7f3aa0000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:42:12.232 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.208+0000 7f3a9cff9640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-29T11:42:12.233 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.209+0000 7f3aa5cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.233 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.209+0000 7f3aa5cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.233 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.209+0000 7f3aa5cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.233 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.209+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a84005080 con 0x7f3aa013f7a0 2024-03-29T11:42:12.233 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.211+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3a98004760 con 0x7f3aa013f7a0 2024-03-29T11:42:12.233 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.211+0000 7f3a9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.233 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.211+0000 7f3a9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:12.257 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:12.255+0000 7fa3002c5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:42:12.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.283+0000 7f3aa5cfb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c466d742-1d0d-4f6f-83c1-9572380d0809", "id": 7}] 2024-03-29T11:42:12.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.283+0000 7f3aa5cfb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c466d742-1d0d-4f6f-83c1-9572380d0809", "id": 7}] 2024-03-29T11:42:12.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.283+0000 7f3aa5cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.283+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3656814900 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c466d742-1d0d-4f6f-83c1-9572380d0809", "id": 7} v 0) v1 -- 0x7f3a840058b0 con 0x7f3aa013f7a0 2024-03-29T11:42:12.299 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.296+0000 7f3a9cff9640 1 -- 172.21.15.171:0/3656814900 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c466d742-1d0d-4f6f-83c1-9572380d0809", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 855494483 0 1371665406) 0x7f3a98006650 con 0x7f3aa013f7a0 2024-03-29T11:42:12.299 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.296+0000 7f3a9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c466d742-1d0d-4f6f-83c1-9572380d0809", "id": 7}] 2024-03-29T11:42:12.299 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.296+0000 7f3a9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:12.299 INFO:teuthology.orchestra.run.smithi171.stdout:7 2024-03-29T11:42:12.300 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f9d20c32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:42:12.301 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 10 librados: watch_flush enter 2024-03-29T11:42:12.301 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 10 librados: watch_flush exit 2024-03-29T11:42:12.301 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-29T11:42:12.301 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 10 client.4156.objecter successfully canceled tick 2024-03-29T11:42:12.301 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 10 monclient: shutdown 2024-03-29T11:42:12.301 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:12.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3656814900 >> v1:172.21.15.171:6789/0 conn(0x7f3aa013f7a0 legacy=0x7f3aa01d7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3656814900 shutdown_connections 2024-03-29T11:42:12.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3656814900 >> 172.21.15.171:0/3656814900 conn(0x7f3aa00b4970 msgr2=0x7f3aa00af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:12.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3656814900 shutdown_connections 2024-03-29T11:42:12.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 1 -- 172.21.15.171:0/3656814900 wait complete. 2024-03-29T11:42:12.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-29T11:42:12.297+0000 7f3aa5cfb640 1 librados: shutdown 2024-03-29T11:42:12.310 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:42:12.311 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:42:12.313 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:42:12.313 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:42:12.313 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:42:12.315 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:42:12.315 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:42:12.315 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:42:12.317 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:42:12.318 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:42:12.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:42:12.320 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:42:12.320 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:42:12.320 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:42:12.322 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:42:12.322 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:42:12.322 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:42:12.324 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:42:12.325 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:42:12.325 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:42:12.327 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:42:12.327 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:42:12.327 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:42:12.329 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:42:12.329 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:12.399 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:12.396+0000 7fa3002c5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:42:12.449 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:12.447+0000 7f9d20c32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:42:12.464 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:12.461+0000 7fa3002c5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:42:12.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.466+0000 7f6d1d433640 10 monclient: get_monmap_and_config 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.466+0000 7f6d1d433640 10 monclient: build_initial_monmap 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.466+0000 7f6d1d433640 10 monclient: monmap: 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 0 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:42:12.467308+0000 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:42:12.467308+0000 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:12.469 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:12.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.466+0000 7f6d1d433640 1 Processor -- start 2024-03-29T11:42:12.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.466+0000 7f6d1d433640 1 -- start start 2024-03-29T11:42:12.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.466+0000 7f6d1d433640 10 monclient: init 2024-03-29T11:42:12.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:12.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 10 monclient(hunting): picked mon.noname-a con 0x7f6d1809ac40 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 10 monclient(hunting): picked mon.noname-c con 0x7f6d1809c870 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 10 monclient(hunting): picked mon.noname-b con 0x7f6d180b4360 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1809dec0 con 0x7f6d1809ac40 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1809da30 con 0x7f6d1809c870 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180b39c0 con 0x7f6d180b4360 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d1d433640 10 monclient(hunting): authenticate will time out at 945.747020s 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d16ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f6d1809ac40 0x7f6d180b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45952/0 (socket says 172.21.15.55:45952) 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d16ffd640 1 -- 172.21.15.55:0/3056914835 learned_addr learned my addr 172.21.15.55:0/3056914835 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d177fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f6d180b4360 0x7f6d1814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:39304/0 (socket says 172.21.15.55:39304) 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217594557 0 0) 0x7f6d08001110 con 0x7f6d1809ac40 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d04003520 con 0x7f6d1809ac40 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885460395 0 0) 0x7f6d180b39c0 con 0x7f6d180b4360 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d08001110 con 0x7f6d180b4360 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242840862 0 0) 0x7f6d1809da30 con 0x7f6d1809c870 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:42:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.467+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d180b39c0 con 0x7f6d1809c870 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 315501300 0 0) 0x7f6d08001110 con 0x7f6d180b4360 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 >> v1:172.21.15.171:6789/0 conn(0x7f6d1809c870 legacy=0x7f6d1813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 >> v1:172.21.15.55:6789/0 conn(0x7f6d1809ac40 legacy=0x7f6d180b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 1 monclient: found mon.noname-b 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d180b5160 con 0x7f6d180b4360 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6d10001920 con 0x7f6d180b4360 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: dump: 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 5 monclient: authenticate success, global_id 4181 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d10001a80 con 0x7f6d180b4360 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 1 -- 172.21.15.55:0/3056914835 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6d10001ed0 con 0x7f6d180b4360 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 4 set_mon_vals no callback set 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d15ffb640 10 monclient: dump: 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 10 monclient: shutdown 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 1 -- 172.21.15.55:0/3056914835 >> v1:172.21.15.55:6790/0 conn(0x7f6d180b4360 legacy=0x7f6d1814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 1 -- 172.21.15.55:0/3056914835 shutdown_connections 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 1 -- 172.21.15.55:0/3056914835 >> 172.21.15.55:0/3056914835 conn(0x7f6d18066b00 msgr2=0x7f6d18066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 1 -- 172.21.15.55:0/3056914835 shutdown_connections 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.468+0000 7f6d1d433640 1 -- 172.21.15.55:0/3056914835 wait complete. 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.469+0000 7f6d1d433640 10 monclient: build_initial_monmap 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.469+0000 7f6d1d433640 10 monclient: monmap: 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 0 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 0.000000 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:created 0.000000 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:12.483 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.469+0000 7f6d1d433640 1 librados: starting msgr at 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.469+0000 7f6d1d433640 1 librados: starting objecter 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.469+0000 7f6d1d433640 1 Processor -- start 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.469+0000 7f6d1d433640 1 -- start start 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.469+0000 7f6d1d433640 1 librados: setting wanted keys 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.469+0000 7f6d1d433640 1 librados: calling monclient init 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.469+0000 7f6d1d433640 10 monclient: init 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 10 monclient(hunting): picked mon.noname-c con 0x7f6d1809ac40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 10 monclient(hunting): picked mon.noname-a con 0x7f6d180ad1e0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:12.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 10 monclient(hunting): picked mon.noname-b con 0x7f6d181434a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180ae320 con 0x7f6d180ad1e0 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d18139850 con 0x7f6d181434a0 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d181d7190 con 0x7f6d1809ac40 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d1d433640 10 monclient(hunting): authenticate will time out at 945.750138s 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d167fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f6d180ad1e0 0x7f6d181d7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45964/0 (socket says 172.21.15.55:45964) 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d167fc640 1 -- 172.21.15.55:0/573160418 learned_addr learned my addr 172.21.15.55:0/573160418 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 client.?.objecter ms_handle_connect 0x7f6d180ad1e0 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 client.?.objecter ms_handle_connect 0x7f6d1809ac40 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 client.?.objecter ms_handle_connect 0x7f6d181434a0 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786891491 0 0) 0x7f6d181d7190 con 0x7f6d1809ac40 2024-03-29T11:42:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf8004eb0 con 0x7f6d1809ac40 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958643961 0 0) 0x7f6d18139850 con 0x7f6d181434a0 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d181d7190 con 0x7f6d181434a0 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770684046 0 0) 0x7f6d180ae320 con 0x7f6d180ad1e0 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.470+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d18139850 con 0x7f6d180ad1e0 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 418795218 0 0) 0x7f6d181d7190 con 0x7f6d181434a0 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d180ae320 con 0x7f6d181434a0 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6d10001740 con 0x7f6d181434a0 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.487 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3232008091 0 0) 0x7f6d18139850 con 0x7f6d180ad1e0 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d181d7190 con 0x7f6d180ad1e0 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 867881259 0 0) 0x7f6cf8004eb0 con 0x7f6d1809ac40 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d18139850 con 0x7f6d1809ac40 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6d00001870 con 0x7f6d180ad1e0 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.488 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6d080012f0 con 0x7f6d1809ac40 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3657331255 0 0) 0x7f6d180ae320 con 0x7f6d181434a0 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 >> v1:172.21.15.55:6790/0 conn(0x7f6d1809ac40 legacy=0x7f6d18139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 >> v1:172.21.15.55:6789/0 conn(0x7f6d180ad1e0 legacy=0x7f6d181d7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 monclient: found mon.b 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d181d7470 con 0x7f6d181434a0 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:12.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d1d433640 5 monclient: authenticate success, global_id 4186 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d1d433640 10 monclient: _renew_subs 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d1d433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d1d433640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d181d80a0 con 0x7f6d181434a0 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d1d433640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d1d433640 10 monclient: _renew_subs 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d1d433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d1d433640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d18145c10 con 0x7f6d181434a0 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d1d433640 1 librados: init done 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d100018a0 con 0x7f6d181434a0 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6d10004fb0 con 0x7f6d181434a0 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 monclient: dump: 2024-03-29T11:42:12.491 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d157fa640 4 set_mon_vals no callback set 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6d100054a0 con 0x7f6d181434a0 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f6d10001420 con 0x7f6d181434a0 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 10 client.4186.objecter ms_dispatch 0x7f6d18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.471+0000 7f6d0f7fe640 20 client.4186.objecter dump_active .. 0 homeless 2024-03-29T11:42:12.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.472+0000 7f6d1d433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.472+0000 7f6d1d433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.472+0000 7f6d1d433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.472+0000 7f6d1d433640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cfc005080 con 0x7f6d181434a0 2024-03-29T11:42:12.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.474+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6d10005f20 con 0x7f6d181434a0 2024-03-29T11:42:12.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.474+0000 7f6d0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.474+0000 7f6d0f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:12.495 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:12.493+0000 7f9d20c32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:42:12.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.536+0000 7f6d1d433640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:12.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.536+0000 7f6d1d433640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:12.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.536+0000 7f6d1d433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:12.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.536+0000 7f6d1d433640 1 -- 172.21.15.55:0/573160418 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cfc005330 con 0x7f6d181434a0 2024-03-29T11:42:12.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.536+0000 7f6d0f7fe640 1 -- 172.21.15.55:0/573160418 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f6d10005750 con 0x7f6d181434a0 2024-03-29T11:42:12.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.536+0000 7f6d0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:12.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.536+0000 7f6d0f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:12.540 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-29T11:42:12.540 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:12.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 10 librados: watch_flush enter 2024-03-29T11:42:12.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 10 librados: watch_flush exit 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 20 client.4186.objecter shutdown clearing up homeless session... 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 10 client.4186.objecter successfully canceled tick 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 10 monclient: shutdown 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 1 -- 172.21.15.55:0/573160418 >> v1:172.21.15.171:6789/0 conn(0x7f6d181434a0 legacy=0x7f6d181e2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 1 -- 172.21.15.55:0/573160418 shutdown_connections 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 1 -- 172.21.15.55:0/573160418 >> 172.21.15.55:0/573160418 conn(0x7f6d18066b00 msgr2=0x7f6d1813a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 1 -- 172.21.15.55:0/573160418 shutdown_connections 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 1 -- 172.21.15.55:0/573160418 wait complete. 2024-03-29T11:42:12.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-29T11:42:12.537+0000 7f6d1d433640 1 librados: shutdown 2024-03-29T11:42:12.551 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:42:12.551 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:42:12.583 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:12.580+0000 7fa3002c5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:42:12.608 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:12.605+0000 7f9d20c32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:42:12.681 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:12.679+0000 7fa3002c5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:42:12.705 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:12.702+0000 7f9d20c32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:42:12.841 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:12.838+0000 7fa3002c5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:42:12.859 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:12.856+0000 7f9d20c32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:42:12.885 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:12.882+0000 7fa3002c5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:42:12.901 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:12.898+0000 7f9d20c32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:42:12.918 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-29T11:42:12.915+0000 7ff5b6a11640 -1 Falling back to public interface 2024-03-29T11:42:12.924 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-03-29T11:42:12.921+0000 7f408fa60640 -1 Falling back to public interface 2024-03-29T11:42:12.924 INFO:tasks.ceph.osd.7.smithi171.stderr:2024-03-29T11:42:12.921+0000 7f406fc19640 -1 Falling back to public interface 2024-03-29T11:42:12.924 INFO:tasks.ceph.osd.7.smithi171.stderr:2024-03-29T11:42:12.921+0000 7f406e605640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2024-03-29T11:42:12.930 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-03-29T11:42:12.927+0000 7fe558eb5640 -1 Falling back to public interface 2024-03-29T11:42:12.939 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-29T11:42:12.936+0000 7fcac240d640 -1 Falling back to public interface 2024-03-29T11:42:12.940 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-29T11:42:12.936+0000 7fcf86e13640 -1 Falling back to public interface 2024-03-29T11:42:12.940 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-29T11:42:12.936+0000 7fcf857d9640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2024-03-29T11:42:12.940 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-29T11:42:12.937+0000 7fcac0d61640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6810/0: (98) Address already in use 2024-03-29T11:42:12.942 INFO:tasks.ceph.osd.6.smithi171.stderr:2024-03-29T11:42:12.939+0000 7f2c7f614640 -1 Falling back to public interface 2024-03-29T11:42:12.945 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-29T11:42:12.942+0000 7eff87f6f640 -1 Falling back to public interface 2024-03-29T11:42:13.046 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:13.043+0000 7fa3002c5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:42:13.059 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:13.056+0000 7f9d20c32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:42:13.358 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:13.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.445+0000 7f2c6b01e640 10 monclient: get_monmap_and_config 2024-03-29T11:42:13.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.445+0000 7f2c6b01e640 10 monclient: build_initial_monmap 2024-03-29T11:42:13.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.445+0000 7f2c6b01e640 10 monclient: monmap: 2024-03-29T11:42:13.448 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:13.448 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:13.449 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:42:13.446847+0000 2024-03-29T11:42:13.449 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:42:13.446847+0000 2024-03-29T11:42:13.449 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:13.449 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:13.449 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:13.449 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:13.449 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:13.449 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.446+0000 7f2c6b01e640 1 Processor -- start 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.446+0000 7f2c6b01e640 1 -- start start 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.446+0000 7f2c6b01e640 10 monclient: init 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.446+0000 7f2c6b01e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.446+0000 7f2c6b01e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.446+0000 7f2c6b01e640 10 monclient(hunting): picked mon.noname-b con 0x7f2c641457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.446+0000 7f2c6b01e640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6413f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.446+0000 7f2c6b01e640 10 monclient(hunting): picked mon.noname-c con 0x7f2c640a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c6b01e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c640aaef0 con 0x7f2c6413f650 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c6b01e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c64145bd0 con 0x7f2c640a9d70 2024-03-29T11:42:13.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c6b01e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c64145eb0 con 0x7f2c641457f0 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c6b01e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c6b01e640 10 monclient(hunting): authenticate will time out at 946.726670s 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c68dba640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f2c641457f0 0x7f2c6413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:39332/0 (socket says 172.21.15.55:39332) 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c68dba640 1 -- 172.21.15.55:0/2176876384 learned_addr learned my addr 172.21.15.55:0/2176876384 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c63fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2c6413f650 0x7f2c640b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45978/0 (socket says 172.21.15.55:45978) 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317137352 0 0) 0x7f2c64145eb0 con 0x7f2c641457f0 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c54003520 con 0x7f2c641457f0 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146622140 0 0) 0x7f2c640aaef0 con 0x7f2c6413f650 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:13.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c64145eb0 con 0x7f2c6413f650 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344227626 0 0) 0x7f2c64145bd0 con 0x7f2c640a9d70 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c640aaef0 con 0x7f2c640a9d70 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291891716 0 0) 0x7f2c64145eb0 con 0x7f2c6413f650 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 >> v1:172.21.15.55:6790/0 conn(0x7f2c641457f0 legacy=0x7f2c6413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 >> v1:172.21.15.171:6789/0 conn(0x7f2c640a9d70 legacy=0x7f2c6414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 monclient: found mon.noname-a 2024-03-29T11:42:13.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c64146340 con 0x7f2c6413f650 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2c50001670 con 0x7f2c6413f650 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: dump: 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:13.456 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c6b01e640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c6b01e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c500017d0 con 0x7f2c6413f650 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 1 -- 172.21.15.55:0/2176876384 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2c50001c20 con 0x7f2c6413f650 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c637fe640 10 monclient: dump: 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.457 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c6b01e640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.447+0000 7f2c6b01e640 4 set_mon_vals no callback set 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.448+0000 7f2c6b01e640 10 monclient: shutdown 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.448+0000 7f2c6b01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.448+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2176876384 >> v1:172.21.15.55:6789/0 conn(0x7f2c6413f650 legacy=0x7f2c640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.448+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2176876384 shutdown_connections 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.448+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2176876384 >> 172.21.15.55:0/2176876384 conn(0x7f2c64066a00 msgr2=0x7f2c64066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:13.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.448+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2176876384 shutdown_connections 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.448+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2176876384 wait complete. 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.449+0000 7f2c6b01e640 10 monclient: build_initial_monmap 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.449+0000 7f2c6b01e640 10 monclient: monmap: 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.450+0000 7f2c6b01e640 1 librados: starting msgr at 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.450+0000 7f2c6b01e640 1 librados: starting objecter 2024-03-29T11:42:13.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.450+0000 7f2c6b01e640 1 Processor -- start 2024-03-29T11:42:13.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.450+0000 7f2c6b01e640 1 -- start start 2024-03-29T11:42:13.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.450+0000 7f2c6b01e640 1 librados: setting wanted keys 2024-03-29T11:42:13.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.450+0000 7f2c6b01e640 1 librados: calling monclient init 2024-03-29T11:42:13.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.450+0000 7f2c6b01e640 10 monclient: init 2024-03-29T11:42:13.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:13.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:13.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 10 monclient(hunting): picked mon.noname-b con 0x7f2c640a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:13.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6413f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:13.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 10 monclient(hunting): picked mon.noname-c con 0x7f2c641457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:13.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c640b1430 con 0x7f2c6413f650 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c640b15c0 con 0x7f2c640a9d70 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c64144840 con 0x7f2c641457f0 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c6b01e640 10 monclient(hunting): authenticate will time out at 946.731052s 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c63fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2c6413f650 0x7f2c64141cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45984/0 (socket says 172.21.15.55:45984) 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c63fff640 1 -- 172.21.15.55:0/2147200688 learned_addr learned my addr 172.21.15.55:0/2147200688 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 client.?.objecter ms_handle_connect 0x7f2c641457f0 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 client.?.objecter ms_handle_connect 0x7f2c6413f650 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 client.?.objecter ms_handle_connect 0x7f2c640a9d70 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630819843 0 0) 0x7f2c640b1430 con 0x7f2c6413f650 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:13.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c48004eb0 con 0x7f2c6413f650 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981355393 0 0) 0x7f2c64144840 con 0x7f2c641457f0 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c640b1430 con 0x7f2c641457f0 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947272742 0 0) 0x7f2c58001090 con 0x7f2c640a9d70 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:13.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c64144840 con 0x7f2c640a9d70 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2231989379 0 0) 0x7f2c48004eb0 con 0x7f2c6413f650 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.451+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c58001090 con 0x7f2c6413f650 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2c50001670 con 0x7f2c6413f650 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): dump: 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:13.467 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3609910086 0 0) 0x7f2c640b1430 con 0x7f2c641457f0 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c48004eb0 con 0x7f2c641457f0 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2c5c001bf0 con 0x7f2c641457f0 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): dump: 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.468 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1280608205 0 0) 0x7f2c64144840 con 0x7f2c640a9d70 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c640b1430 con 0x7f2c640a9d70 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2c58001790 con 0x7f2c640a9d70 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient(hunting): dump: 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.469 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3485961929 0 0) 0x7f2c58001090 con 0x7f2c6413f650 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 >> v1:172.21.15.55:6790/0 conn(0x7f2c641457f0 legacy=0x7f2c64142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 >> v1:172.21.15.171:6789/0 conn(0x7f2c640a9d70 legacy=0x7f2c641d7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 monclient: found mon.a 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c641449d0 con 0x7f2c6413f650 2024-03-29T11:42:13.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c6b01e640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c6b01e640 10 monclient: _renew_subs 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c6b01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c641e3030 con 0x7f2c6413f650 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c6b01e640 10 client.4194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c6b01e640 10 monclient: _renew_subs 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c6b01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c641e33a0 con 0x7f2c6413f650 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c6b01e640 1 librados: init done 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c50004a90 con 0x7f2c6413f650 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2c50001230 con 0x7f2c6413f650 2024-03-29T11:42:13.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c62ffd640 4 set_mon_vals no callback set 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 monclient: dump: 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2c50005270 con 0x7f2c6413f650 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f2c50006420 con 0x7f2c6413f650 2024-03-29T11:42:13.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 10 client.4194.objecter ms_dispatch 0x7f2c64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 3 client.4194.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 3 client.4194.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.452+0000 7f2c617fa640 20 client.4194.objecter dump_active .. 0 homeless 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.453+0000 7f2c6b01e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.453+0000 7f2c6b01e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.453+0000 7f2c6b01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.453+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c44005080 con 0x7f2c6413f650 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.455+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2c50006630 con 0x7f2c6413f650 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.455+0000 7f2c617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:13.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.455+0000 7f2c617fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:13.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.518+0000 7f2c6b01e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:13.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.518+0000 7f2c6b01e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:13.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.518+0000 7f2c6b01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:42:13.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.518+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2147200688 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c44005330 con 0x7f2c6413f650 2024-03-29T11:42:13.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.518+0000 7f2c617fa640 1 -- 172.21.15.55:0/2147200688 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f2c500068d0 con 0x7f2c6413f650 2024-03-29T11:42:13.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.518+0000 7f2c617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:13.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.518+0000 7f2c617fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:13.522 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:42:13.522 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:13.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 10 librados: watch_flush enter 2024-03-29T11:42:13.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 10 librados: watch_flush exit 2024-03-29T11:42:13.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 20 client.4194.objecter shutdown clearing up homeless session... 2024-03-29T11:42:13.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 10 client.4194.objecter successfully canceled tick 2024-03-29T11:42:13.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 10 monclient: shutdown 2024-03-29T11:42:13.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:13.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2147200688 >> v1:172.21.15.55:6789/0 conn(0x7f2c6413f650 legacy=0x7f2c64141cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2147200688 shutdown_connections 2024-03-29T11:42:13.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2147200688 >> 172.21.15.55:0/2147200688 conn(0x7f2c64066a00 msgr2=0x7f2c640b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:13.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2147200688 shutdown_connections 2024-03-29T11:42:13.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 1 -- 172.21.15.55:0/2147200688 wait complete. 2024-03-29T11:42:13.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:13.519+0000 7f2c6b01e640 1 librados: shutdown 2024-03-29T11:42:13.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:13.880 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:13.877+0000 7f9d20c32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:42:13.886 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:13.884+0000 7fa3002c5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:42:14.155 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:14.151+0000 7f9d20c32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:42:14.173 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:14.169+0000 7fa3002c5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:42:14.206 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:14.202+0000 7f9d20c32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:42:14.225 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:14.222+0000 7fa3002c5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:42:14.322 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:14.318+0000 7f9d20c32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:42:14.346 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-29T11:42:14.343+0000 7fa3002c5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:42:15.335 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:15.331+0000 7f9cc11ae640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:42:15.335 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-29T11:42:15.332+0000 7f9cc11ae640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:42:16.390 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-03-29T11:42:16.386+0000 7f408fa60640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:42:16.390 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-29T11:42:16.387+0000 7ff5b6a11640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:42:16.435 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-29T11:42:16.432+0000 7fcac240d640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:42:16.438 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-29T11:42:16.435+0000 7fcf86e13640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:42:16.444 INFO:tasks.ceph.osd.7.smithi171.stderr:2024-03-29T11:42:16.440+0000 7f406fc19640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:42:16.444 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-03-29T11:42:16.440+0000 7fe558eb5640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:42:16.461 INFO:tasks.ceph.osd.6.smithi171.stderr:2024-03-29T11:42:16.457+0000 7f2c7f614640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:42:16.476 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-29T11:42:16.473+0000 7eff87f6f640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:42:20.338 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:20.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.483+0000 7ffb9bbc8640 10 monclient: get_monmap_and_config 2024-03-29T11:42:20.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.483+0000 7ffb9bbc8640 10 monclient: build_initial_monmap 2024-03-29T11:42:20.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.483+0000 7ffb9bbc8640 10 monclient: monmap: 2024-03-29T11:42:20.486 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:20.487 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:20.487 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:42:20.484475+0000 2024-03-29T11:42:20.487 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:42:20.484475+0000 2024-03-29T11:42:20.487 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:20.487 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:20.487 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:20.487 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:20.487 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:20.487 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.484+0000 7ffb9bbc8640 1 Processor -- start 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.484+0000 7ffb9bbc8640 1 -- start start 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.484+0000 7ffb9bbc8640 10 monclient: init 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 10 monclient(hunting): picked mon.noname-a con 0x7ffb940b0770 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 10 monclient(hunting): picked mon.noname-c con 0x7ffb9414a700 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 10 monclient(hunting): picked mon.noname-b con 0x7ffb940ac5a0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940aba70 con 0x7ffb940b0770 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940aecb0 con 0x7ffb9414a700 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940af150 con 0x7ffb940ac5a0 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:20.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb9bbc8640 10 monclient(hunting): authenticate will time out at 953.764954s 2024-03-29T11:42:20.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb99163640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ffb9414a700 0x7ffb940abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:33430/0 (socket says 172.21.15.55:33430) 2024-03-29T11:42:20.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.485+0000 7ffb99163640 1 -- 172.21.15.55:0/2956664904 learned_addr learned my addr 172.21.15.55:0/2956664904 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711413427 0 0) 0x7ffb940aba70 con 0x7ffb940b0770 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb80003520 con 0x7ffb940b0770 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537209237 0 0) 0x7ffb940aecb0 con 0x7ffb9414a700 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb940aba70 con 0x7ffb9414a700 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1097234955 0 0) 0x7ffb80003520 con 0x7ffb940b0770 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.486+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 >> v1:172.21.15.55:6790/0 conn(0x7ffb940ac5a0 legacy=0x7ffb94151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 >> v1:172.21.15.171:6789/0 conn(0x7ffb9414a700 legacy=0x7ffb940abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: _finish_hunting 0 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 1 monclient: found mon.noname-a 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940ac9d0 con 0x7ffb940b0770 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: _finish_auth 0 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: _check_auth_tickets 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ffb90001690 con 0x7ffb940b0770 2024-03-29T11:42:20.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: dump: 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb9bbc8640 5 monclient: authenticate success, global_id 4239 2024-03-29T11:42:20.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb9bbc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:20.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb900017f0 con 0x7ffb940b0770 2024-03-29T11:42:20.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:20.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 1 -- 172.21.15.55:0/2956664904 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ffb90001c00 con 0x7ffb940b0770 2024-03-29T11:42:20.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:20.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.487+0000 7ffb98962640 10 monclient: dump: 2024-03-29T11:42:20.496 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:20.496 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:20.496 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.488+0000 7ffb9bbc8640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.488+0000 7ffb9bbc8640 4 set_mon_vals no callback set 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.488+0000 7ffb9bbc8640 10 monclient: shutdown 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.488+0000 7ffb9bbc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.488+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/2956664904 >> v1:172.21.15.55:6789/0 conn(0x7ffb940b0770 legacy=0x7ffb94149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.489+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/2956664904 shutdown_connections 2024-03-29T11:42:20.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.489+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/2956664904 >> 172.21.15.55:0/2956664904 conn(0x7ffb9406b660 msgr2=0x7ffb9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.489+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/2956664904 shutdown_connections 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.489+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/2956664904 wait complete. 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.490+0000 7ffb9bbc8640 10 monclient: build_initial_monmap 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.490+0000 7ffb9bbc8640 10 monclient: monmap: 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:20.498 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.490+0000 7ffb9bbc8640 1 librados: starting msgr at 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.490+0000 7ffb9bbc8640 1 librados: starting objecter 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.490+0000 7ffb9bbc8640 1 Processor -- start 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 1 -- start start 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 1 librados: setting wanted keys 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 1 librados: calling monclient init 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 10 monclient: init 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 10 monclient(hunting): picked mon.noname-a con 0x7ffb940ac5a0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 10 monclient(hunting): picked mon.noname-b con 0x7ffb940b0770 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:20.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 10 monclient(hunting): picked mon.noname-c con 0x7ffb9414a700 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb941d9300 con 0x7ffb940ac5a0 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb94071b40 con 0x7ffb940b0770 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940715d0 con 0x7ffb9414a700 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.491+0000 7ffb9bbc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb9bbc8640 10 monclient(hunting): authenticate will time out at 953.771647s 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb99163640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ffb940b0770 0x7ffb941dba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:33572/0 (socket says 172.21.15.55:33572) 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb9a165640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7ffb9414a700 0x7ffb941e6fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:54040/0 (socket says 172.21.15.55:54040) 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb99163640 1 -- 172.21.15.55:0/210471540 learned_addr learned my addr 172.21.15.55:0/210471540 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 client.?.objecter ms_handle_connect 0x7ffb940b0770 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 client.?.objecter ms_handle_connect 0x7ffb940ac5a0 2024-03-29T11:42:20.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 client.?.objecter ms_handle_connect 0x7ffb9414a700 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500118396 0 0) 0x7ffb94071b40 con 0x7ffb940b0770 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb78004eb0 con 0x7ffb940b0770 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84006175 0 0) 0x7ffb940715d0 con 0x7ffb9414a700 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.492+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb94071b40 con 0x7ffb9414a700 2024-03-29T11:42:20.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233077914 0 0) 0x7ffb941d9300 con 0x7ffb940ac5a0 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb940715d0 con 0x7ffb940ac5a0 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3865791515 0 0) 0x7ffb78004eb0 con 0x7ffb940b0770 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb941d9300 con 0x7ffb940b0770 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ffb84001950 con 0x7ffb940b0770 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:20.502 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3333370344 0 0) 0x7ffb940715d0 con 0x7ffb940ac5a0 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb78004eb0 con 0x7ffb940ac5a0 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ffb90001b00 con 0x7ffb940ac5a0 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:20.503 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:20.504 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.504 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.504 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.504 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:20.504 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:20.504 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:20.504 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:20.504 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1247164417 0 0) 0x7ffb941d9300 con 0x7ffb940b0770 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 >> v1:172.21.15.55:6790/0 conn(0x7ffb9414a700 legacy=0x7ffb941e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 >> v1:172.21.15.55:6789/0 conn(0x7ffb940ac5a0 legacy=0x7ffb94142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 monclient: found mon.b 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94071760 con 0x7ffb940b0770 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb9bbc8640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:42:20.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb9bbc8640 10 monclient: _renew_subs 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb9bbc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb94124320 con 0x7ffb940b0770 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb9bbc8640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb9bbc8640 10 monclient: _renew_subs 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb9bbc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb94142920 con 0x7ffb940b0770 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.493+0000 7ffb9bbc8640 1 librados: init done 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.494+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb840047e0 con 0x7ffb940b0770 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.494+0000 7ffb8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.494+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ffb84004af0 con 0x7ffb940b0770 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.494+0000 7ffb8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.494+0000 7ffb8a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.494+0000 7ffb8a7fc640 10 monclient: dump: 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:20.507 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.494+0000 7ffb8bfff640 4 set_mon_vals no callback set 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.495+0000 7ffb9bbc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.497+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7ffb8401aac0 con 0x7ffb940b0770 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.498+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7ffb84084af0 con 0x7ffb940b0770 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.498+0000 7ffb8a7fc640 10 client.4237.objecter ms_dispatch 0x7ffb94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.498+0000 7ffb8a7fc640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:20.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.498+0000 7ffb8a7fc640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:20.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.498+0000 7ffb8a7fc640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T11:42:20.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.498+0000 7ffb9bbc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:20.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.498+0000 7ffb9bbc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:20.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.498+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb7c005080 con 0x7ffb940b0770 2024-03-29T11:42:20.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.499+0000 7ffb99964640 10 monclient: get_auth_request con 0x7ffb7806dff0 auth_method 0 2024-03-29T11:42:20.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.499+0000 7ffb8a7fc640 10 client.4237.objecter ms_handle_connect 0x7ffb7806dff0 2024-03-29T11:42:20.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.506+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb84050200 con 0x7ffb940b0770 2024-03-29T11:42:20.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.506+0000 7ffb8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:20.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.506+0000 7ffb8a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:20.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.877+0000 7ffb9bbc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:20.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.877+0000 7ffb9bbc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:20.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.877+0000 7ffb9bbc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:20.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.877+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb7c005330 con 0x7ffb940b0770 2024-03-29T11:42:20.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.877+0000 7ffb8a7fc640 1 -- 172.21.15.55:0/210471540 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7ffb84052300 con 0x7ffb940b0770 2024-03-29T11:42:20.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.877+0000 7ffb8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:20.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.877+0000 7ffb8a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:20.881 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:42:20.881 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:20.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 10 librados: watch_flush enter 2024-03-29T11:42:20.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 10 librados: watch_flush exit 2024-03-29T11:42:20.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T11:42:20.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 10 client.4237.objecter successfully canceled tick 2024-03-29T11:42:20.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 >> v1:172.21.15.171:6816/1463197163 conn(0x7ffb7806dff0 legacy=0x7ffb780704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 10 monclient: shutdown 2024-03-29T11:42:20.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:20.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 >> v1:172.21.15.171:6789/0 conn(0x7ffb940b0770 legacy=0x7ffb941dba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 shutdown_connections 2024-03-29T11:42:20.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 >> 172.21.15.55:0/210471540 conn(0x7ffb9406b660 msgr2=0x7ffb940abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:20.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.880+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 shutdown_connections 2024-03-29T11:42:20.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.881+0000 7ffb9bbc8640 1 -- 172.21.15.55:0/210471540 wait complete. 2024-03-29T11:42:20.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:20.881+0000 7ffb9bbc8640 1 librados: shutdown 2024-03-29T11:42:20.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:27.705 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:27.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.845+0000 7fa8248de640 10 monclient: get_monmap_and_config 2024-03-29T11:42:27.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.845+0000 7fa8248de640 10 monclient: build_initial_monmap 2024-03-29T11:42:27.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.845+0000 7fa8248de640 10 monclient: monmap: 2024-03-29T11:42:27.848 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:27.848 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:27.848 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:42:27.846199+0000 2024-03-29T11:42:27.848 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:42:27.846199+0000 2024-03-29T11:42:27.849 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:27.849 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:27.849 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:27.849 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:27.849 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:27.849 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:27.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.847+0000 7fa8248de640 1 Processor -- start 2024-03-29T11:42:27.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.847+0000 7fa8248de640 1 -- start start 2024-03-29T11:42:27.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.847+0000 7fa8248de640 10 monclient: init 2024-03-29T11:42:27.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:27.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 10 monclient(hunting): picked mon.noname-b con 0x7fa818006e30 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 10 monclient(hunting): picked mon.noname-c con 0x7fa818035e00 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 10 monclient(hunting): picked mon.noname-a con 0x7fa8180c9670 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa818035970 con 0x7fa8180c9670 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa818036280 con 0x7fa818035e00 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa818035640 con 0x7fa818006e30 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa81f7fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fa818006e30 0x7fa8180cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:55838/0 (socket says 172.21.15.55:55838) 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa8248de640 10 monclient(hunting): authenticate will time out at 961.127944s 2024-03-29T11:42:27.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa81f7fe640 1 -- 172.21.15.55:0/973431759 learned_addr learned my addr 172.21.15.55:0/973431759 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359513382 0 0) 0x7fa818035640 con 0x7fa818006e30 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa81e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa81e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa81e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.848+0000 7fa81e7fc640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa810003520 con 0x7fa818006e30 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89643827 0 0) 0x7fa818036280 con 0x7fa818035e00 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa818035640 con 0x7fa818035e00 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688361815 0 0) 0x7fa818035640 con 0x7fa818035e00 2024-03-29T11:42:27.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 >> v1:172.21.15.55:6790/0 conn(0x7fa818006e30 legacy=0x7fa8180cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 >> v1:172.21.15.55:6789/0 conn(0x7fa8180c9670 legacy=0x7fa8180d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa818034e20 con 0x7fa818035e00 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa814001690 con 0x7fa818035e00 2024-03-29T11:42:27.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient: dump: 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa8248de640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa8248de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:27.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa814001860 con 0x7fa818035e00 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa81e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa8248de640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.849+0000 7fa8248de640 4 set_mon_vals no callback set 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa81e7fc640 1 -- 172.21.15.55:0/973431759 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa814001cb0 con 0x7fa818035e00 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa81e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa81e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa81e7fc640 10 monclient: dump: 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:27.858 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:27.859 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:27.859 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:27.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa8248de640 10 monclient: shutdown 2024-03-29T11:42:27.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa8248de640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:27.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa8248de640 1 -- 172.21.15.55:0/973431759 >> v1:172.21.15.171:6789/0 conn(0x7fa818035e00 legacy=0x7fa8180c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa8248de640 1 -- 172.21.15.55:0/973431759 shutdown_connections 2024-03-29T11:42:27.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa8248de640 1 -- 172.21.15.55:0/973431759 >> 172.21.15.55:0/973431759 conn(0x7fa818005c00 msgr2=0x7fa818006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:27.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa8248de640 1 -- 172.21.15.55:0/973431759 shutdown_connections 2024-03-29T11:42:27.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.850+0000 7fa8248de640 1 -- 172.21.15.55:0/973431759 wait complete. 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.851+0000 7fa8248de640 10 monclient: build_initial_monmap 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.851+0000 7fa8248de640 10 monclient: monmap: 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:27.867 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.851+0000 7fa8248de640 1 librados: starting msgr at 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.851+0000 7fa8248de640 1 librados: starting objecter 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.852+0000 7fa8248de640 1 Processor -- start 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.852+0000 7fa8248de640 1 -- start start 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.852+0000 7fa8248de640 1 librados: setting wanted keys 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.852+0000 7fa8248de640 1 librados: calling monclient init 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.852+0000 7fa8248de640 10 monclient: init 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.852+0000 7fa8248de640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.852+0000 7fa8248de640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa8248de640 10 monclient(hunting): picked mon.noname-c con 0x7fa818006e30 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa8248de640 10 monclient(hunting): picked mon.noname-a con 0x7fa818035e00 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa8248de640 10 monclient(hunting): picked mon.noname-b con 0x7fa8180c9670 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa8248de640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa818162040 con 0x7fa818035e00 2024-03-29T11:42:27.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa8248de640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81802ff10 con 0x7fa8180c9670 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa8248de640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81802ee20 con 0x7fa818006e30 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa8248de640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa8248de640 10 monclient(hunting): authenticate will time out at 961.132734s 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa81effd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa818035e00 0x7fa81802f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:40068/0 (socket says 172.21.15.55:40068) 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa81effd640 1 -- 172.21.15.55:0/2348541890 learned_addr learned my addr 172.21.15.55:0/2348541890 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 client.?.objecter ms_handle_connect 0x7fa818035e00 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 client.?.objecter ms_handle_connect 0x7fa8180c9670 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425568130 0 0) 0x7fa818162040 con 0x7fa818035e00 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:27.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa804004eb0 con 0x7fa818035e00 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070172628 0 0) 0x7fa81802ff10 con 0x7fa8180c9670 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.853+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa818162040 con 0x7fa8180c9670 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 client.?.objecter ms_handle_connect 0x7fa818006e30 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2277848211 0 0) 0x7fa818162040 con 0x7fa8180c9670 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa81802ff10 con 0x7fa8180c9670 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa80c001820 con 0x7fa8180c9670 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:27.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient(hunting): dump: 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531265464 0 0) 0x7fa81802ee20 con 0x7fa818006e30 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:27.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa818162040 con 0x7fa818006e30 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 165748352 0 0) 0x7fa804004eb0 con 0x7fa818035e00 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa804005ab0 con 0x7fa818035e00 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 161109252 0 0) 0x7fa81802ff10 con 0x7fa8180c9670 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 >> v1:172.21.15.55:6790/0 conn(0x7fa818006e30 legacy=0x7fa818161930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 >> v1:172.21.15.55:6789/0 conn(0x7fa818035e00 legacy=0x7fa81802f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 monclient: found mon.b 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81802f100 con 0x7fa8180c9670 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient: _finish_auth 0 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:27.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa80bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa8248de640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa8248de640 10 monclient: _renew_subs 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.854+0000 7fa8248de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.855+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa818161130 con 0x7fa8180c9670 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.855+0000 7fa8248de640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.855+0000 7fa8248de640 10 monclient: _renew_subs 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.855+0000 7fa8248de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.855+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81802e080 con 0x7fa8180c9670 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.855+0000 7fa8248de640 1 librados: init done 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.856+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa80c001d10 con 0x7fa8180c9670 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.856+0000 7fa80bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.856+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa80c003790 con 0x7fa8180c9670 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.856+0000 7fa80bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:27.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.856+0000 7fa80bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.856+0000 7fa80bfff640 10 monclient: dump: 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.856+0000 7fa81dffb640 4 set_mon_vals no callback set 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.856+0000 7fa8248de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.858+0000 7fa8248de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:27.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.858+0000 7fa8248de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:27.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.858+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa800005080 con 0x7fa8180c9670 2024-03-29T11:42:27.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.858+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fa80c019760 con 0x7fa8180c9670 2024-03-29T11:42:27.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.859+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fa80c0836d0 con 0x7fa8180c9670 2024-03-29T11:42:27.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.859+0000 7fa80bfff640 10 client.4243.objecter ms_dispatch 0x7fa820000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:27.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.859+0000 7fa80bfff640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:27.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.859+0000 7fa80bfff640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:27.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.859+0000 7fa80bfff640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T11:42:27.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.859+0000 7fa81f7fe640 10 monclient: get_auth_request con 0x7fa80406e180 auth_method 0 2024-03-29T11:42:27.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.860+0000 7fa80bfff640 10 client.4243.objecter ms_handle_connect 0x7fa80406e180 2024-03-29T11:42:27.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.868+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa80c04efb0 con 0x7fa8180c9670 2024-03-29T11:42:27.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.868+0000 7fa80bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:27.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:27.868+0000 7fa80bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:28.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.240+0000 7fa8248de640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:28.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.240+0000 7fa8248de640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:28.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.240+0000 7fa8248de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:28.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.240+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa800005330 con 0x7fa8180c9670 2024-03-29T11:42:28.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.242+0000 7fa80bfff640 1 -- 172.21.15.55:0/2348541890 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fa80c050ee0 con 0x7fa8180c9670 2024-03-29T11:42:28.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.242+0000 7fa80bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:28.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.242+0000 7fa80bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:28.245 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:42:28.246 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:28.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.245+0000 7fa8248de640 10 librados: watch_flush enter 2024-03-29T11:42:28.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 10 librados: watch_flush exit 2024-03-29T11:42:28.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T11:42:28.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 10 client.4243.objecter successfully canceled tick 2024-03-29T11:42:28.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 >> v1:172.21.15.171:6816/1463197163 conn(0x7fa80406e180 legacy=0x7fa804070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 10 monclient: shutdown 2024-03-29T11:42:28.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:28.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 >> v1:172.21.15.171:6789/0 conn(0x7fa8180c9670 legacy=0x7fa81802d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 shutdown_connections 2024-03-29T11:42:28.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 >> 172.21.15.55:0/2348541890 conn(0x7fa818005c00 msgr2=0x7fa8180c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:28.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 shutdown_connections 2024-03-29T11:42:28.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 1 -- 172.21.15.55:0/2348541890 wait complete. 2024-03-29T11:42:28.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:28.246+0000 7fa8248de640 1 librados: shutdown 2024-03-29T11:42:28.262 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:35.069 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:35.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.215+0000 7fe27da84640 10 monclient: get_monmap_and_config 2024-03-29T11:42:35.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.215+0000 7fe27da84640 10 monclient: build_initial_monmap 2024-03-29T11:42:35.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.215+0000 7fe27da84640 10 monclient: monmap: 2024-03-29T11:42:35.219 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:42:35.216317+0000 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:42:35.216317+0000 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.216+0000 7fe27da84640 1 Processor -- start 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.216+0000 7fe27da84640 1 -- start start 2024-03-29T11:42:35.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.216+0000 7fe27da84640 10 monclient: init 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 10 monclient(hunting): picked mon.noname-c con 0x7fe2780a9930 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 10 monclient(hunting): picked mon.noname-b con 0x7fe278149dc0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 10 monclient(hunting): picked mon.noname-a con 0x7fe27814ff40 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2780aae60 con 0x7fe27814ff40 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2780b4310 con 0x7fe2780a9930 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2780afdd0 con 0x7fe278149dc0 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:35.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.217+0000 7fe27da84640 10 monclient(hunting): authenticate will time out at 968.497190s 2024-03-29T11:42:35.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.219+0000 7fe2777fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe2780a9930 0x7fe278143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:42036/0 (socket says 172.21.15.55:42036) 2024-03-29T11:42:35.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.219+0000 7fe2777fe640 1 -- 172.21.15.55:0/2563024851 learned_addr learned my addr 172.21.15.55:0/2563024851 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.220+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036208529 0 0) 0x7fe2780afdd0 con 0x7fe278149dc0 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe260003520 con 0x7fe278149dc0 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241481861 0 0) 0x7fe2780b4310 con 0x7fe2780a9930 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2780afdd0 con 0x7fe2780a9930 2024-03-29T11:42:35.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293420722 0 0) 0x7fe2780aae60 con 0x7fe27814ff40 2024-03-29T11:42:35.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:35.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:35.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:35.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:42:35.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.221+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2780b4310 con 0x7fe27814ff40 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766973301 0 0) 0x7fe260003520 con 0x7fe278149dc0 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 >> v1:172.21.15.171:6789/0 conn(0x7fe2780a9930 legacy=0x7fe278143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 >> v1:172.21.15.55:6789/0 conn(0x7fe27814ff40 legacy=0x7fe278152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 1 monclient: found mon.noname-b 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27814a950 con 0x7fe278149dc0 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:35.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe2680015c0 con 0x7fe278149dc0 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: dump: 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe268001720 con 0x7fe278149dc0 2024-03-29T11:42:35.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 1 -- 172.21.15.55:0/2563024851 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe268001b70 con 0x7fe278149dc0 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.222+0000 7fe2767fc640 10 monclient: dump: 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:35.228 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:35.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.224+0000 7fe27da84640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:42:35.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.224+0000 7fe27da84640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:35.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.224+0000 7fe27da84640 4 set_mon_vals no callback set 2024-03-29T11:42:35.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.225+0000 7fe27da84640 10 monclient: shutdown 2024-03-29T11:42:35.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.225+0000 7fe27da84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:35.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.225+0000 7fe27da84640 1 -- 172.21.15.55:0/2563024851 >> v1:172.21.15.55:6790/0 conn(0x7fe278149dc0 legacy=0x7fe27814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:35.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.225+0000 7fe27da84640 1 -- 172.21.15.55:0/2563024851 shutdown_connections 2024-03-29T11:42:35.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.225+0000 7fe27da84640 1 -- 172.21.15.55:0/2563024851 >> 172.21.15.55:0/2563024851 conn(0x7fe2780b35e0 msgr2=0x7fe2780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.225+0000 7fe27da84640 1 -- 172.21.15.55:0/2563024851 shutdown_connections 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.225+0000 7fe27da84640 1 -- 172.21.15.55:0/2563024851 wait complete. 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.226+0000 7fe27da84640 10 monclient: build_initial_monmap 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.226+0000 7fe27da84640 10 monclient: monmap: 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:35.230 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:35.231 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:35.231 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:35.231 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:35.231 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:35.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.226+0000 7fe27da84640 1 librados: starting msgr at 2024-03-29T11:42:35.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.226+0000 7fe27da84640 1 librados: starting objecter 2024-03-29T11:42:35.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.226+0000 7fe27da84640 1 Processor -- start 2024-03-29T11:42:35.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.227+0000 7fe27da84640 1 -- start start 2024-03-29T11:42:35.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.227+0000 7fe27da84640 1 librados: setting wanted keys 2024-03-29T11:42:35.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.227+0000 7fe27da84640 1 librados: calling monclient init 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.227+0000 7fe27da84640 10 monclient: init 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.227+0000 7fe27da84640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe27da84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe27da84640 10 monclient(hunting): picked mon.noname-b con 0x7fe2780a9930 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe27da84640 10 monclient(hunting): picked mon.noname-a con 0x7fe278149dc0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe27da84640 10 monclient(hunting): picked mon.noname-c con 0x7fe27814ff40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe27da84640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27811bb40 con 0x7fe278149dc0 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe27da84640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2781db8d0 con 0x7fe2780a9930 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe27da84640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2781db4f0 con 0x7fe27814ff40 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe27da84640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe27da84640 10 monclient(hunting): authenticate will time out at 968.507737s 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe2777fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe2780a9930 0x7fe27806f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:42312/0 (socket says 172.21.15.55:42312) 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe2777fe640 1 -- 172.21.15.55:0/2243035462 learned_addr learned my addr 172.21.15.55:0/2243035462 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:35.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe25ffff640 10 client.?.objecter ms_handle_connect 0x7fe2780a9930 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe25ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234661296 0 0) 0x7fe2781db8d0 con 0x7fe2780a9930 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe25ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe25ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe25ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe25ffff640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.228+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe264004eb0 con 0x7fe2780a9930 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.229+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3610389270 0 0) 0x7fe264004eb0 con 0x7fe2780a9930 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.229+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2781db8d0 con 0x7fe2780a9930 2024-03-29T11:42:35.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.229+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe26c001930 con 0x7fe2780a9930 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.229+0000 7fe25ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.229+0000 7fe25ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.229+0000 7fe25ffff640 10 monclient(hunting): dump: 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.229+0000 7fe25ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:35.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.229+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1846304689 0 0) 0x7fe2781db8d0 con 0x7fe2780a9930 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.229+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 >> v1:172.21.15.55:6790/0 conn(0x7fe27814ff40 legacy=0x7fe27811b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 >> v1:172.21.15.55:6789/0 conn(0x7fe278149dc0 legacy=0x7fe2781dbdf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe25ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe25ffff640 1 monclient: found mon.b 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe25ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe25ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2781db680 con 0x7fe2780a9930 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe25ffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe25ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe25ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe27da84640 5 monclient: authenticate success, global_id 4249 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe27da84640 10 monclient: _renew_subs 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe27da84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:35.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe278071df0 con 0x7fe2780a9930 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe27da84640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe27da84640 10 monclient: _renew_subs 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe27da84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27813d310 con 0x7fe2780a9930 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.232+0000 7fe27da84640 1 librados: init done 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.233+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe26c001a90 con 0x7fe2780a9930 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.233+0000 7fe25ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.233+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe26c004c30 con 0x7fe2780a9930 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.233+0000 7fe25ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.233+0000 7fe25ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.233+0000 7fe25ffff640 10 monclient: dump: 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:35.241 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:35.242 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:35.242 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:35.242 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:35.242 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:35.242 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:35.242 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:35.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.233+0000 7fe275ffb640 4 set_mon_vals no callback set 2024-03-29T11:42:35.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.235+0000 7fe27da84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:35.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.235+0000 7fe27da84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:35.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.235+0000 7fe27da84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:35.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.235+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe258005080 con 0x7fe2780a9930 2024-03-29T11:42:35.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.236+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fe26c004d90 con 0x7fe2780a9930 2024-03-29T11:42:35.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.237+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fe26c084bb0 con 0x7fe2780a9930 2024-03-29T11:42:35.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.237+0000 7fe25ffff640 10 client.4249.objecter ms_dispatch 0x7fe278000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:35.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.237+0000 7fe25ffff640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:35.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.237+0000 7fe25ffff640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:35.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.237+0000 7fe25ffff640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-29T11:42:35.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.237+0000 7fe276ffd640 10 monclient: get_auth_request con 0x7fe26406dbd0 auth_method 0 2024-03-29T11:42:35.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.238+0000 7fe25ffff640 10 client.4249.objecter ms_handle_connect 0x7fe26406dbd0 2024-03-29T11:42:35.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.243+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe26c0502c0 con 0x7fe2780a9930 2024-03-29T11:42:35.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.243+0000 7fe25ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:35.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.243+0000 7fe25ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:35.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.584+0000 7fe27da84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:35.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.584+0000 7fe27da84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:35.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.584+0000 7fe27da84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.584+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe258005330 con 0x7fe2780a9930 2024-03-29T11:42:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.585+0000 7fe25ffff640 1 -- 172.21.15.55:0/2243035462 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fe26c0523c0 con 0x7fe2780a9930 2024-03-29T11:42:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.585+0000 7fe25ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.585+0000 7fe25ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:35.589 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:42:35.589 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.589+0000 7fe27da84640 10 librados: watch_flush enter 2024-03-29T11:42:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 10 librados: watch_flush exit 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 10 client.4249.objecter successfully canceled tick 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 >> v1:172.21.15.171:6816/1463197163 conn(0x7fe26406dbd0 legacy=0x7fe264070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 10 monclient: shutdown 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 >> v1:172.21.15.171:6789/0 conn(0x7fe2780a9930 legacy=0x7fe27806f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 shutdown_connections 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 >> 172.21.15.55:0/2243035462 conn(0x7fe2780b35e0 msgr2=0x7fe278140a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 shutdown_connections 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 1 -- 172.21.15.55:0/2243035462 wait complete. 2024-03-29T11:42:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:35.590+0000 7fe27da84640 1 librados: shutdown 2024-03-29T11:42:35.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:42.416 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.573+0000 7f898278d640 10 monclient: get_monmap_and_config 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.573+0000 7f898278d640 10 monclient: build_initial_monmap 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.573+0000 7f898278d640 10 monclient: monmap: 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:42:42.574611+0000 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:42:42.574611+0000 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:42.578 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:42.579 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:42.579 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:42.579 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:42.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.575+0000 7f898278d640 1 Processor -- start 2024-03-29T11:42:42.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.575+0000 7f898278d640 1 -- start start 2024-03-29T11:42:42.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.575+0000 7f898278d640 10 monclient: init 2024-03-29T11:42:42.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:42.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:42.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 10 monclient(hunting): picked mon.noname-b con 0x7f897c0a9ac0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:42.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 10 monclient(hunting): picked mon.noname-a con 0x7f897c13ee90 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:42.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 10 monclient(hunting): picked mon.noname-c con 0x7f897c0af900 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:42.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897c0afd10 con 0x7f897c13ee90 2024-03-29T11:42:42.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897c0af6f0 con 0x7f897c0af900 2024-03-29T11:42:42.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897c0aac80 con 0x7f897c0a9ac0 2024-03-29T11:42:42.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:42.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.576+0000 7f898278d640 10 monclient(hunting): authenticate will time out at 975.856104s 2024-03-29T11:42:42.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.580+0000 7f897bfff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f897c0a9ac0 0x7f897c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:37501/0 (socket says 172.21.15.55:37501) 2024-03-29T11:42:42.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.580+0000 7f897bfff640 1 -- 172.21.15.55:0/2649264682 learned_addr learned my addr 172.21.15.55:0/2649264682 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:42.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.581+0000 7f897affd640 1 -- 172.21.15.55:0/2649264682 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400004953 0 0) 0x7f897c0aac80 con 0x7f897c0a9ac0 2024-03-29T11:42:42.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:42.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:42.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:42.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:42:42.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 1 -- 172.21.15.55:0/2649264682 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8964003520 con 0x7f897c0a9ac0 2024-03-29T11:42:42.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 1 -- 172.21.15.55:0/2649264682 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2180350192 0 0) 0x7f8964003520 con 0x7f897c0a9ac0 2024-03-29T11:42:42.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 1 -- 172.21.15.55:0/2649264682 >> v1:172.21.15.171:6789/0 conn(0x7f897c0af900 legacy=0x7f897c149b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 1 -- 172.21.15.55:0/2649264682 >> v1:172.21.15.55:6789/0 conn(0x7f897c13ee90 legacy=0x7f897c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 1 monclient: found mon.noname-b 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.582+0000 7f897affd640 1 -- 172.21.15.55:0/2649264682 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c0aaf60 con 0x7f897c0a9ac0 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f897affd640 10 monclient: _finish_auth 0 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f897affd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f897affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f898278d640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f898278d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f897affd640 1 -- 172.21.15.55:0/2649264682 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8974001690 con 0x7f897c0a9ac0 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f897affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:42.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f897affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f897affd640 10 monclient: dump: 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f898278d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f897affd640 1 -- 172.21.15.55:0/2649264682 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8974001920 con 0x7f897c0a9ac0 2024-03-29T11:42:42.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f897affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:42.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f898278d640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:42.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.583+0000 7f898278d640 4 set_mon_vals no callback set 2024-03-29T11:42:42.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.584+0000 7f898278d640 10 monclient: shutdown 2024-03-29T11:42:42.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.584+0000 7f898278d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:42.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.584+0000 7f898278d640 1 -- 172.21.15.55:0/2649264682 >> v1:172.21.15.55:6790/0 conn(0x7f897c0a9ac0 legacy=0x7f897c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:42.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.585+0000 7f898278d640 1 -- 172.21.15.55:0/2649264682 shutdown_connections 2024-03-29T11:42:42.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.585+0000 7f898278d640 1 -- 172.21.15.55:0/2649264682 >> 172.21.15.55:0/2649264682 conn(0x7f897c0b35e0 msgr2=0x7f897c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.585+0000 7f898278d640 1 -- 172.21.15.55:0/2649264682 shutdown_connections 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.585+0000 7f898278d640 1 -- 172.21.15.55:0/2649264682 wait complete. 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.586+0000 7f898278d640 10 monclient: build_initial_monmap 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.586+0000 7f898278d640 10 monclient: monmap: 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:42.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.586+0000 7f898278d640 1 librados: starting msgr at 2024-03-29T11:42:42.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.586+0000 7f898278d640 1 librados: starting objecter 2024-03-29T11:42:42.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.587+0000 7f898278d640 1 Processor -- start 2024-03-29T11:42:42.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.587+0000 7f898278d640 1 -- start start 2024-03-29T11:42:42.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.587+0000 7f898278d640 1 librados: setting wanted keys 2024-03-29T11:42:42.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.587+0000 7f898278d640 1 librados: calling monclient init 2024-03-29T11:42:42.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.587+0000 7f898278d640 10 monclient: init 2024-03-29T11:42:42.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.588+0000 7f898278d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:42.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.588+0000 7f898278d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.590+0000 7f898278d640 10 monclient(hunting): picked mon.noname-c con 0x7f897c0a9ac0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.590+0000 7f898278d640 10 monclient(hunting): picked mon.noname-a con 0x7f897c0af900 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.590+0000 7f898278d640 10 monclient(hunting): picked mon.noname-b con 0x7f897c13ee90 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.590+0000 7f898278d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897c13fcd0 con 0x7f897c0af900 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.590+0000 7f898278d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897c1318c0 con 0x7f897c13ee90 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.590+0000 7f898278d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897c142070 con 0x7f897c0a9ac0 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.590+0000 7f898278d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.590+0000 7f898278d640 10 monclient(hunting): authenticate will time out at 975.870061s 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.592+0000 7f897bfff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f897c0a9ac0 0x7f897c13f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:37567/0 (socket says 172.21.15.55:37567) 2024-03-29T11:42:42.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.592+0000 7f897bfff640 1 -- 172.21.15.55:0/619446851 learned_addr learned my addr 172.21.15.55:0/619446851 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 client.?.objecter ms_handle_connect 0x7f897c0a9ac0 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 client.?.objecter ms_handle_connect 0x7f897c0af900 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 client.?.objecter ms_handle_connect 0x7f897c13ee90 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306741513 0 0) 0x7f897c142070 con 0x7f897c0a9ac0 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:42.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8968004eb0 con 0x7f897c0a9ac0 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243356584 0 0) 0x7f897c13fcd0 con 0x7f897c0af900 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f897c142070 con 0x7f897c0af900 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969715209 0 0) 0x7f897c1318c0 con 0x7f897c13ee90 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:42:42.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.597+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f897c13fcd0 con 0x7f897c13ee90 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.598+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 777506634 0 0) 0x7f897c13fcd0 con 0x7f897c13ee90 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.598+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f897c1318c0 con 0x7f897c13ee90 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.598+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f896c0019e0 con 0x7f897c13ee90 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.598+0000 7f8978ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.598+0000 7f8978ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.598+0000 7f8978ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:42.608 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.598+0000 7f8978ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.598+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221207904 0 0) 0x7f897c142070 con 0x7f897c0af900 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f897c13fcd0 con 0x7f897c0af900 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f89700016e0 con 0x7f897c0af900 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:42.609 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2790212502 0 0) 0x7f8968004eb0 con 0x7f897c0a9ac0 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f897c142070 con 0x7f897c0a9ac0 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f89740049c0 con 0x7f897c0a9ac0 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:42.610 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 612388443 0 0) 0x7f897c142070 con 0x7f897c0a9ac0 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.599+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 >> v1:172.21.15.171:6789/0 conn(0x7f897c13ee90 legacy=0x7f897c1311b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 >> v1:172.21.15.55:6789/0 conn(0x7f897c0af900 legacy=0x7f897c130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f8978ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f8978ff9640 1 monclient: found mon.c 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f8978ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f8978ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c142200 con 0x7f897c0a9ac0 2024-03-29T11:42:42.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f8978ff9640 10 monclient: _finish_auth 0 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f8978ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f8978ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f898278d640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f898278d640 10 monclient: _renew_subs 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f898278d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897c1df730 con 0x7f897c0a9ac0 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f898278d640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f898278d640 10 monclient: _renew_subs 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f898278d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f897c06e7a0 con 0x7f897c0a9ac0 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.600+0000 7f898278d640 1 librados: init done 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.601+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8974004f00 con 0x7f897c0a9ac0 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.601+0000 7f8978ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.601+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8974005290 con 0x7f897c0a9ac0 2024-03-29T11:42:42.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.601+0000 7f8978ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.601+0000 7f8978ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.601+0000 7f8978ff9640 10 monclient: dump: 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.602+0000 7f898278d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:42.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.602+0000 7f898278d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:42.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.602+0000 7f898278d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:42.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.602+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897c0aa900 con 0x7f897c0a9ac0 2024-03-29T11:42:42.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.602+0000 7f8979ffb640 4 set_mon_vals no callback set 2024-03-29T11:42:42.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.604+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f8974001b50 con 0x7f897c0a9ac0 2024-03-29T11:42:42.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.605+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f8974084cc0 con 0x7f897c0a9ac0 2024-03-29T11:42:42.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.605+0000 7f8978ff9640 10 client.4250.objecter ms_dispatch 0x7f897c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:42.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.605+0000 7f8978ff9640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:42.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.605+0000 7f8978ff9640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:42.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.605+0000 7f8978ff9640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T11:42:42.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.609+0000 7f897b7fe640 10 monclient: get_auth_request con 0x7f896806e220 auth_method 0 2024-03-29T11:42:42.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.610+0000 7f8978ff9640 10 client.4250.objecter ms_handle_connect 0x7f896806e220 2024-03-29T11:42:42.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.616+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89740505a0 con 0x7f897c0a9ac0 2024-03-29T11:42:42.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.616+0000 7f8978ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:42.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.616+0000 7f8978ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:42.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.941+0000 7f898278d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:42.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.941+0000 7f898278d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:42.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.941+0000 7f898278d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:42.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.941+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f897c075f10 con 0x7f897c0a9ac0 2024-03-29T11:42:42.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.945+0000 7f8978ff9640 1 -- 172.21.15.55:0/619446851 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f89740524d0 con 0x7f897c0a9ac0 2024-03-29T11:42:42.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.945+0000 7f8978ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:42.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.945+0000 7f8978ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:42.948 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:42:42.948 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:42.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.948+0000 7f898278d640 10 librados: watch_flush enter 2024-03-29T11:42:42.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.948+0000 7f898278d640 10 librados: watch_flush exit 2024-03-29T11:42:42.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.948+0000 7f898278d640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T11:42:42.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.948+0000 7f898278d640 10 client.4250.objecter successfully canceled tick 2024-03-29T11:42:42.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.948+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 >> v1:172.21.15.171:6816/1463197163 conn(0x7f896806e220 legacy=0x7f89680706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:42.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.949+0000 7f898278d640 10 monclient: shutdown 2024-03-29T11:42:42.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.949+0000 7f898278d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:42.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.949+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 >> v1:172.21.15.55:6790/0 conn(0x7f897c0a9ac0 legacy=0x7f897c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:42.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.949+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 shutdown_connections 2024-03-29T11:42:42.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.949+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 >> 172.21.15.55:0/619446851 conn(0x7f897c0b35e0 msgr2=0x7f897c0b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:42.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.949+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 shutdown_connections 2024-03-29T11:42:42.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.949+0000 7f898278d640 1 -- 172.21.15.55:0/619446851 wait complete. 2024-03-29T11:42:42.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:42.949+0000 7f898278d640 1 librados: shutdown 2024-03-29T11:42:42.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:49.776 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:49.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.922+0000 7fb32eccf640 10 monclient: get_monmap_and_config 2024-03-29T11:42:49.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.922+0000 7fb32eccf640 10 monclient: build_initial_monmap 2024-03-29T11:42:49.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.922+0000 7fb32eccf640 10 monclient: monmap: 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:42:49.923451+0000 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:42:49.923451+0000 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:49.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.923+0000 7fb32eccf640 1 Processor -- start 2024-03-29T11:42:49.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.923+0000 7fb32eccf640 1 -- start start 2024-03-29T11:42:49.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.923+0000 7fb32eccf640 10 monclient: init 2024-03-29T11:42:49.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:49.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:49.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 10 monclient(hunting): picked mon.noname-b con 0x7fb3280a9ac0 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:49.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 10 monclient(hunting): picked mon.noname-a con 0x7fb32813ee90 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:49.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 10 monclient(hunting): picked mon.noname-c con 0x7fb3280af900 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:49.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280afd10 con 0x7fb32813ee90 2024-03-29T11:42:49.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280af6f0 con 0x7fb3280af900 2024-03-29T11:42:49.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280aac80 con 0x7fb3280a9ac0 2024-03-29T11:42:49.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:49.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.924+0000 7fb32eccf640 10 monclient(hunting): authenticate will time out at 983.204198s 2024-03-29T11:42:49.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.929+0000 7fb327fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fb32813ee90 0x7fb32813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:36879/0 (socket says 172.21.15.55:36879) 2024-03-29T11:42:49.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb327fff640 1 -- 172.21.15.55:0/1448334848 learned_addr learned my addr 172.21.15.55:0/1448334848 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:49.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 1 -- 172.21.15.55:0/1448334848 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021151932 0 0) 0x7fb3280aac80 con 0x7fb3280a9ac0 2024-03-29T11:42:49.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:49.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:49.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:49.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:42:49.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 1 -- 172.21.15.55:0/1448334848 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb310003520 con 0x7fb3280a9ac0 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 1 -- 172.21.15.55:0/1448334848 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2614424285 0 0) 0x7fb310003520 con 0x7fb3280a9ac0 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 1 -- 172.21.15.55:0/1448334848 >> v1:172.21.15.171:6789/0 conn(0x7fb3280af900 legacy=0x7fb328149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 1 -- 172.21.15.55:0/1448334848 >> v1:172.21.15.55:6789/0 conn(0x7fb32813ee90 legacy=0x7fb32813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 1 monclient: found mon.noname-b 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.930+0000 7fb3277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 1 -- 172.21.15.55:0/1448334848 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3280aaf60 con 0x7fb3280a9ac0 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 1 -- 172.21.15.55:0/1448334848 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fb320001a10 con 0x7fb3280a9ac0 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: dump: 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb32eccf640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb32eccf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 1 -- 172.21.15.55:0/1448334848 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb320001be0 con 0x7fb3280a9ac0 2024-03-29T11:42:49.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 1 -- 172.21.15.55:0/1448334848 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fb320001ff0 con 0x7fb3280a9ac0 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb3277fe640 10 monclient: dump: 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:49.941 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb32eccf640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.931+0000 7fb32eccf640 4 set_mon_vals no callback set 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.932+0000 7fb32eccf640 10 monclient: shutdown 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.932+0000 7fb32eccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.932+0000 7fb32eccf640 1 -- 172.21.15.55:0/1448334848 >> v1:172.21.15.55:6790/0 conn(0x7fb3280a9ac0 legacy=0x7fb3280adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.932+0000 7fb32eccf640 1 -- 172.21.15.55:0/1448334848 shutdown_connections 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.932+0000 7fb32eccf640 1 -- 172.21.15.55:0/1448334848 >> 172.21.15.55:0/1448334848 conn(0x7fb3280b35e0 msgr2=0x7fb3280b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:49.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.933+0000 7fb32eccf640 1 -- 172.21.15.55:0/1448334848 shutdown_connections 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.933+0000 7fb32eccf640 1 -- 172.21.15.55:0/1448334848 wait complete. 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.934+0000 7fb32eccf640 10 monclient: build_initial_monmap 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.934+0000 7fb32eccf640 10 monclient: monmap: 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.934+0000 7fb32eccf640 1 librados: starting msgr at 2024-03-29T11:42:49.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.934+0000 7fb32eccf640 1 librados: starting objecter 2024-03-29T11:42:49.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.934+0000 7fb32eccf640 1 Processor -- start 2024-03-29T11:42:49.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.934+0000 7fb32eccf640 1 -- start start 2024-03-29T11:42:49.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.934+0000 7fb32eccf640 1 librados: setting wanted keys 2024-03-29T11:42:49.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.934+0000 7fb32eccf640 1 librados: calling monclient init 2024-03-29T11:42:49.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.934+0000 7fb32eccf640 10 monclient: init 2024-03-29T11:42:49.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.935+0000 7fb32eccf640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:49.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.936+0000 7fb32eccf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:49.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.936+0000 7fb32eccf640 10 monclient(hunting): picked mon.noname-a con 0x7fb3280a9ac0 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.936+0000 7fb32eccf640 10 monclient(hunting): picked mon.noname-c con 0x7fb3280af900 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.937+0000 7fb32eccf640 10 monclient(hunting): picked mon.noname-b con 0x7fb32813ee90 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.937+0000 7fb32eccf640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32814b130 con 0x7fb3280a9ac0 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.937+0000 7fb32eccf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32814c320 con 0x7fb32813ee90 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.937+0000 7fb32eccf640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32814c590 con 0x7fb3280af900 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.937+0000 7fb32eccf640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.937+0000 7fb32eccf640 10 monclient(hunting): authenticate will time out at 983.217219s 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.939+0000 7fb327fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fb3280af900 0x7fb3281e7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:40083/0 (socket says 172.21.15.55:40083) 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.939+0000 7fb327fff640 1 -- 172.21.15.55:0/338606497 learned_addr learned my addr 172.21.15.55:0/338606497 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.939+0000 7fb3257fa640 10 client.?.objecter ms_handle_connect 0x7fb3280af900 2024-03-29T11:42:49.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.939+0000 7fb3257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.942+0000 7fb3257fa640 10 client.?.objecter ms_handle_connect 0x7fb3280a9ac0 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.942+0000 7fb3257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.942+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91497909 0 0) 0x7fb32814c590 con 0x7fb3280af900 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.942+0000 7fb3257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.942+0000 7fb3257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.942+0000 7fb3257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.942+0000 7fb3257fa640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.942+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb314004eb0 con 0x7fb3280af900 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.943+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 227930322 0 0) 0x7fb314004eb0 con 0x7fb3280af900 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.943+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb32814c590 con 0x7fb3280af900 2024-03-29T11:42:49.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.943+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fb318001060 con 0x7fb3280af900 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.943+0000 7fb3257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.943+0000 7fb3257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.943+0000 7fb3257fa640 10 monclient(hunting): dump: 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:49.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 10 client.?.objecter ms_handle_connect 0x7fb32813ee90 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2347565396 0 0) 0x7fb32814c590 con 0x7fb3280af900 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 >> v1:172.21.15.171:6789/0 conn(0x7fb32813ee90 legacy=0x7fb3281e8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 >> v1:172.21.15.55:6789/0 conn(0x7fb3280a9ac0 legacy=0x7fb3281e4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 1 monclient: found mon.c 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32814b5c0 con 0x7fb3280af900 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.944+0000 7fb3257fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.945+0000 7fb3257fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.945+0000 7fb3257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.945+0000 7fb32eccf640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.945+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb318001570 con 0x7fb3280af900 2024-03-29T11:42:49.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.945+0000 7fb32eccf640 10 monclient: _renew_subs 2024-03-29T11:42:49.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb32eccf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3280723c0 con 0x7fb3280af900 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb32eccf640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb32eccf640 10 monclient: _renew_subs 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb32eccf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32814f210 con 0x7fb3280af900 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb32eccf640 1 librados: init done 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb3257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fb318001900 con 0x7fb3280af900 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb3257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb3257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.946+0000 7fb3257fa640 10 monclient: dump: 2024-03-29T11:42:49.954 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.947+0000 7fb326ffd640 4 set_mon_vals no callback set 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.947+0000 7fb32eccf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.948+0000 7fb32eccf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.948+0000 7fb32eccf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:49.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.948+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb308005080 con 0x7fb3280af900 2024-03-29T11:42:49.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.951+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fb318001a60 con 0x7fb3280af900 2024-03-29T11:42:49.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.952+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fb318001d10 con 0x7fb3280af900 2024-03-29T11:42:49.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.952+0000 7fb3257fa640 10 client.4256.objecter ms_dispatch 0x7fb328000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:49.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.952+0000 7fb3257fa640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:49.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.952+0000 7fb3257fa640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:49.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.952+0000 7fb3257fa640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T11:42:49.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.952+0000 7fb32ca6b640 10 monclient: get_auth_request con 0x7fb31406db60 auth_method 0 2024-03-29T11:42:49.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.953+0000 7fb3257fa640 10 client.4256.objecter ms_handle_connect 0x7fb31406db60 2024-03-29T11:42:49.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.961+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb318050af0 con 0x7fb3280af900 2024-03-29T11:42:49.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.961+0000 7fb3257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:49.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:49.961+0000 7fb3257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:50.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.300+0000 7fb32eccf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:50.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.300+0000 7fb32eccf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:50.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.300+0000 7fb32eccf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:42:50.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.300+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb308005330 con 0x7fb3280af900 2024-03-29T11:42:50.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.301+0000 7fb3257fa640 1 -- 172.21.15.55:0/338606497 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fb318050af0 con 0x7fb3280af900 2024-03-29T11:42:50.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.301+0000 7fb3257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:50.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.301+0000 7fb3257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:50.307 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:42:50.307 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:50.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.304+0000 7fb32eccf640 10 librados: watch_flush enter 2024-03-29T11:42:50.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.304+0000 7fb32eccf640 10 librados: watch_flush exit 2024-03-29T11:42:50.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.304+0000 7fb32eccf640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T11:42:50.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.304+0000 7fb32eccf640 10 client.4256.objecter successfully canceled tick 2024-03-29T11:42:50.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.304+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 >> v1:172.21.15.171:6816/1463197163 conn(0x7fb31406db60 legacy=0x7fb314070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.304+0000 7fb32eccf640 10 monclient: shutdown 2024-03-29T11:42:50.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.304+0000 7fb32eccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:50.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.304+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 >> v1:172.21.15.55:6790/0 conn(0x7fb3280af900 legacy=0x7fb3281e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.305+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 shutdown_connections 2024-03-29T11:42:50.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.305+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 >> 172.21.15.55:0/338606497 conn(0x7fb3280b35e0 msgr2=0x7fb3280ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:50.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.305+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 shutdown_connections 2024-03-29T11:42:50.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.305+0000 7fb32eccf640 1 -- 172.21.15.55:0/338606497 wait complete. 2024-03-29T11:42:50.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:50.305+0000 7fb32eccf640 1 librados: shutdown 2024-03-29T11:42:50.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:57.132 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:57.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.285+0000 7fdb43fff640 10 monclient: get_monmap_and_config 2024-03-29T11:42:57.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.285+0000 7fdb43fff640 10 monclient: build_initial_monmap 2024-03-29T11:42:57.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.285+0000 7fdb43fff640 10 monclient: monmap: 2024-03-29T11:42:57.288 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:57.288 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:57.288 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:42:57.286025+0000 2024-03-29T11:42:57.288 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:42:57.286025+0000 2024-03-29T11:42:57.288 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:57.289 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:57.289 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:57.289 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:42:57.289 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:42:57.289 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.287+0000 7fdb43fff640 1 Processor -- start 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.287+0000 7fdb43fff640 1 -- start start 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.287+0000 7fdb43fff640 10 monclient: init 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 10 monclient(hunting): picked mon.noname-a con 0x7fdb440a9750 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 10 monclient(hunting): picked mon.noname-b con 0x7fdb440add40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 10 monclient(hunting): picked mon.noname-c con 0x7fdb440aad20 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb44149870 con 0x7fdb440a9750 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb440abf80 con 0x7fdb440aad20 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb44149ac0 con 0x7fdb440add40 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.288+0000 7fdb43fff640 10 monclient(hunting): authenticate will time out at 990.567185s 2024-03-29T11:42:57.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.292+0000 7fdb42ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fdb440a9750 0x7fdb44149fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58489/0 (socket says 172.21.15.55:58489) 2024-03-29T11:42:57.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.292+0000 7fdb42ffd640 1 -- 172.21.15.55:0/1128198008 learned_addr learned my addr 172.21.15.55:0/1128198008 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:57.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025272074 0 0) 0x7fdb440abf80 con 0x7fdb440aad20 2024-03-29T11:42:57.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:57.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:57.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:57.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:42:57.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb2c003520 con 0x7fdb440aad20 2024-03-29T11:42:57.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188530142 0 0) 0x7fdb44149870 con 0x7fdb440a9750 2024-03-29T11:42:57.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb440abf80 con 0x7fdb440a9750 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258509700 0 0) 0x7fdb44149ac0 con 0x7fdb440add40 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb44149870 con 0x7fdb440add40 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2352182485 0 0) 0x7fdb2c003520 con 0x7fdb440aad20 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 >> v1:172.21.15.55:6790/0 conn(0x7fdb440add40 legacy=0x7fdb44143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 >> v1:172.21.15.55:6789/0 conn(0x7fdb440a9750 legacy=0x7fdb44149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:57.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 monclient: found mon.noname-c 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4414a6f0 con 0x7fdb440aad20 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdb340015c0 con 0x7fdb440aad20 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.293+0000 7fdb41ffb640 10 monclient: dump: 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:57.297 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:57.298 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:57.298 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:57.298 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:57.298 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:57.298 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:57.298 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:57.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.296+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb340017e0 con 0x7fdb440aad20 2024-03-29T11:42:57.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.296+0000 7fdb41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:57.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.296+0000 7fdb41ffb640 1 -- 172.21.15.55:0/1128198008 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdb34001b70 con 0x7fdb440aad20 2024-03-29T11:42:57.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.296+0000 7fdb41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:57.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.296+0000 7fdb41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:57.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.296+0000 7fdb41ffb640 10 monclient: dump: 2024-03-29T11:42:57.299 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:57.299 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.296+0000 7fdb43fff640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.296+0000 7fdb43fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:57.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.296+0000 7fdb43fff640 4 set_mon_vals no callback set 2024-03-29T11:42:57.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.297+0000 7fdb43fff640 10 monclient: shutdown 2024-03-29T11:42:57.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.297+0000 7fdb43fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:57.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.297+0000 7fdb43fff640 1 -- 172.21.15.55:0/1128198008 >> v1:172.21.15.171:6789/0 conn(0x7fdb440aad20 legacy=0x7fdb44151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:57.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.298+0000 7fdb43fff640 1 -- 172.21.15.55:0/1128198008 shutdown_connections 2024-03-29T11:42:57.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.298+0000 7fdb43fff640 1 -- 172.21.15.55:0/1128198008 >> 172.21.15.55:0/1128198008 conn(0x7fdb440673e0 msgr2=0x7fdb440b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:57.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.301+0000 7fdb43fff640 1 -- 172.21.15.55:0/1128198008 shutdown_connections 2024-03-29T11:42:57.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.302+0000 7fdb43fff640 1 -- 172.21.15.55:0/1128198008 wait complete. 2024-03-29T11:42:57.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.303+0000 7fdb43fff640 10 monclient: build_initial_monmap 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.303+0000 7fdb43fff640 10 monclient: monmap: 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.303+0000 7fdb43fff640 1 librados: starting msgr at 2024-03-29T11:42:57.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.303+0000 7fdb43fff640 1 librados: starting objecter 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 1 Processor -- start 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 1 -- start start 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 1 librados: setting wanted keys 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 1 librados: calling monclient init 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 10 monclient: init 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 10 monclient(hunting): picked mon.noname-b con 0x7fdb440a9750 addr v1:172.21.15.171:6789/0 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 10 monclient(hunting): picked mon.noname-a con 0x7fdb440aad20 addr v1:172.21.15.55:6789/0 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 10 monclient(hunting): picked mon.noname-c con 0x7fdb440add40 addr v1:172.21.15.55:6790/0 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb440707a0 con 0x7fdb440aad20 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb44070a80 con 0x7fdb440a9750 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb44070d60 con 0x7fdb440add40 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:57.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.304+0000 7fdb43fff640 10 monclient(hunting): authenticate will time out at 990.583617s 2024-03-29T11:42:57.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.309+0000 7fdb42ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdb440a9750 0x7fdb4406ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:51647/0 (socket says 172.21.15.55:51647) 2024-03-29T11:42:57.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.309+0000 7fdb42ffd640 1 -- 172.21.15.55:0/2196824513 learned_addr learned my addr 172.21.15.55:0/2196824513 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:42:57.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.311+0000 7fdb2b7fe640 10 client.?.objecter ms_handle_connect 0x7fdb440a9750 2024-03-29T11:42:57.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.311+0000 7fdb2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:57.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.311+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286371111 0 0) 0x7fdb44070a80 con 0x7fdb440a9750 2024-03-29T11:42:57.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.311+0000 7fdb2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.311+0000 7fdb2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.311+0000 7fdb2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.311+0000 7fdb2b7fe640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.311+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb30004eb0 con 0x7fdb440a9750 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 client.?.objecter ms_handle_connect 0x7fdb440add40 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 644973381 0 0) 0x7fdb30004eb0 con 0x7fdb440a9750 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb30005300 con 0x7fdb440a9750 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdb3c001dd0 con 0x7fdb440a9750 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:57.317 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439661468 0 0) 0x7fdb44070d60 con 0x7fdb440add40 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:57.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb30005850 con 0x7fdb440add40 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.312+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3670141631 0 0) 0x7fdb30005300 con 0x7fdb440a9750 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 >> v1:172.21.15.55:6790/0 conn(0x7fdb440add40 legacy=0x7fdb441ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 >> v1:172.21.15.55:6789/0 conn(0x7fdb440aad20 legacy=0x7fdb44147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 1 monclient: found mon.b 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb440724a0 con 0x7fdb440a9750 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb3c0020a0 con 0x7fdb440a9750 2024-03-29T11:42:57.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.313+0000 7fdb2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:57.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.314+0000 7fdb417fa640 4 set_mon_vals no callback set 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdb3c001dd0 con 0x7fdb440a9750 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb2b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb2b7fe640 10 monclient: dump: 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:42:57.322 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb43fff640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb43fff640 10 monclient: _renew_subs 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb43fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb43fff640 1 -- 172.21.15.55:0/2196824513 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb44072190 con 0x7fdb440a9750 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb43fff640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb43fff640 10 monclient: _renew_subs 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb43fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb43fff640 1 -- 172.21.15.55:0/2196824513 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4406e870 con 0x7fdb440a9750 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.315+0000 7fdb43fff640 1 librados: init done 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.317+0000 7fdb297fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.317+0000 7fdb297fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.317+0000 7fdb297fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:57.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.317+0000 7fdb297fa640 1 -- 172.21.15.55:0/2196824513 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb440726e0 con 0x7fdb440a9750 2024-03-29T11:42:57.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.321+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fdb3c01ad00 con 0x7fdb440a9750 2024-03-29T11:42:57.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.321+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fdb3c084a60 con 0x7fdb440a9750 2024-03-29T11:42:57.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.321+0000 7fdb2b7fe640 10 client.4261.objecter ms_dispatch 0x7fdb44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:57.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.321+0000 7fdb2b7fe640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:57.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.321+0000 7fdb2b7fe640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:57.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.321+0000 7fdb2b7fe640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:42:57.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.322+0000 7fdb427fc640 10 monclient: get_auth_request con 0x7fdb3006e2b0 auth_method 0 2024-03-29T11:42:57.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.322+0000 7fdb2b7fe640 10 client.4261.objecter ms_handle_connect 0x7fdb3006e2b0 2024-03-29T11:42:57.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.328+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb3c050170 con 0x7fdb440a9750 2024-03-29T11:42:57.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.328+0000 7fdb2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:57.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.328+0000 7fdb2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:57.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.658+0000 7fdb297fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:57.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.658+0000 7fdb297fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:57.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.658+0000 7fdb297fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:42:57.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.658+0000 7fdb297fa640 1 -- 172.21.15.55:0/2196824513 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb440726e0 con 0x7fdb440a9750 2024-03-29T11:42:57.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.662+0000 7fdb2b7fe640 1 -- 172.21.15.55:0/2196824513 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fdb440726e0 con 0x7fdb440a9750 2024-03-29T11:42:57.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.662+0000 7fdb2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:57.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.662+0000 7fdb2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:57.670 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:42:57.670 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:57.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.668+0000 7fdb297fa640 10 librados: watch_flush enter 2024-03-29T11:42:57.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.669+0000 7fdb297fa640 10 librados: watch_flush exit 2024-03-29T11:42:57.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.669+0000 7fdb297fa640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:42:57.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.669+0000 7fdb297fa640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:42:57.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.671+0000 7fdb297fa640 1 -- 172.21.15.55:0/2196824513 >> v1:172.21.15.171:6816/1463197163 conn(0x7fdb3006e2b0 legacy=0x7fdb30070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:57.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.671+0000 7fdb297fa640 10 monclient: shutdown 2024-03-29T11:42:57.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.671+0000 7fdb297fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:57.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.671+0000 7fdb297fa640 1 -- 172.21.15.55:0/2196824513 >> v1:172.21.15.171:6789/0 conn(0x7fdb440a9750 legacy=0x7fdb4406ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:57.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.671+0000 7fdb297fa640 1 -- 172.21.15.55:0/2196824513 shutdown_connections 2024-03-29T11:42:57.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.671+0000 7fdb297fa640 1 -- 172.21.15.55:0/2196824513 >> 172.21.15.55:0/2196824513 conn(0x7fdb440673e0 msgr2=0x7fdb440b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:57.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.671+0000 7fdb297fa640 1 -- 172.21.15.55:0/2196824513 shutdown_connections 2024-03-29T11:42:57.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.671+0000 7fdb297fa640 1 -- 172.21.15.55:0/2196824513 wait complete. 2024-03-29T11:42:57.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:42:57.671+0000 7fdb297fa640 1 librados: shutdown 2024-03-29T11:42:57.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:04.500 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:04.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.652+0000 7f0755225640 10 monclient: get_monmap_and_config 2024-03-29T11:43:04.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.652+0000 7f0755225640 10 monclient: build_initial_monmap 2024-03-29T11:43:04.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.652+0000 7f0755225640 10 monclient: monmap: 2024-03-29T11:43:04.655 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:04.655 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:04.656 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:43:04.653271+0000 2024-03-29T11:43:04.656 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:43:04.653271+0000 2024-03-29T11:43:04.656 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:04.656 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:04.656 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:04.656 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:43:04.656 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:43:04.656 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:04.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.655+0000 7f0755225640 1 Processor -- start 2024-03-29T11:43:04.659 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.655+0000 7f0755225640 1 -- start start 2024-03-29T11:43:04.659 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.655+0000 7f0755225640 10 monclient: init 2024-03-29T11:43:04.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.656+0000 7f0755225640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:04.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.656+0000 7f0755225640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:04.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.657+0000 7f0755225640 10 monclient(hunting): picked mon.noname-b con 0x7f07501457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:04.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.657+0000 7f0755225640 10 monclient(hunting): picked mon.noname-a con 0x7f075013f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:04.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f074ed9d640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f07501457f0 0x7f075013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:39017/0 (socket says 172.21.15.55:39017) 2024-03-29T11:43:04.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f0755225640 10 monclient(hunting): picked mon.noname-c con 0x7f07500a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:04.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f0755225640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500aaef0 con 0x7f075013f650 2024-03-29T11:43:04.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f0755225640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0750145bd0 con 0x7f07500a9d70 2024-03-29T11:43:04.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f0755225640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0750145eb0 con 0x7f07501457f0 2024-03-29T11:43:04.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f0755225640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:04.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f0755225640 10 monclient(hunting): authenticate will time out at 997.939868s 2024-03-29T11:43:04.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f074ed9d640 1 -- 172.21.15.55:0/2027344974 learned_addr learned my addr 172.21.15.55:0/2027344974 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:04.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f074dd9b640 1 -- 172.21.15.55:0/2027344974 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372828778 0 0) 0x7f0750145eb0 con 0x7f07501457f0 2024-03-29T11:43:04.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f074dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:43:04.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f074dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:04.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f074dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:04.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f074dd9b640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:43:04.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.660+0000 7f074dd9b640 1 -- 172.21.15.55:0/2027344974 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0738003520 con 0x7f07501457f0 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 1 -- 172.21.15.55:0/2027344974 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1014222458 0 0) 0x7f0738003520 con 0x7f07501457f0 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 1 -- 172.21.15.55:0/2027344974 >> v1:172.21.15.171:6789/0 conn(0x7f07500a9d70 legacy=0x7f075014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 1 -- 172.21.15.55:0/2027344974 >> v1:172.21.15.55:6789/0 conn(0x7f075013f650 legacy=0x7f07500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 1 monclient: found mon.noname-b 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 1 -- 172.21.15.55:0/2027344974 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0750146340 con 0x7f07501457f0 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:43:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 1 -- 172.21.15.55:0/2027344974 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0748001a10 con 0x7f07501457f0 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.661+0000 7f074dd9b640 10 monclient: dump: 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:04.670 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f0755225640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f0755225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f074dd9b640 1 -- 172.21.15.55:0/2027344974 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0748001be0 con 0x7f07501457f0 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f074dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f074dd9b640 1 -- 172.21.15.55:0/2027344974 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0748002030 con 0x7f07501457f0 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f074dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f074dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f074dd9b640 10 monclient: dump: 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:04.671 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f0755225640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.664+0000 7f0755225640 4 set_mon_vals no callback set 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.665+0000 7f0755225640 10 monclient: shutdown 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.665+0000 7f0755225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.665+0000 7f0755225640 1 -- 172.21.15.55:0/2027344974 >> v1:172.21.15.55:6790/0 conn(0x7f07501457f0 legacy=0x7f075013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.665+0000 7f0755225640 1 -- 172.21.15.55:0/2027344974 shutdown_connections 2024-03-29T11:43:04.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.665+0000 7f0755225640 1 -- 172.21.15.55:0/2027344974 >> 172.21.15.55:0/2027344974 conn(0x7f0750066a00 msgr2=0x7f0750066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:04.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.668+0000 7f0755225640 1 -- 172.21.15.55:0/2027344974 shutdown_connections 2024-03-29T11:43:04.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.668+0000 7f0755225640 1 -- 172.21.15.55:0/2027344974 wait complete. 2024-03-29T11:43:04.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.669+0000 7f0755225640 10 monclient: build_initial_monmap 2024-03-29T11:43:04.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.669+0000 7f0755225640 10 monclient: monmap: 2024-03-29T11:43:04.675 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:04.675 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:04.675 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.670+0000 7f0755225640 1 librados: starting msgr at 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.670+0000 7f0755225640 1 librados: starting objecter 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.670+0000 7f0755225640 1 Processor -- start 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.670+0000 7f0755225640 1 -- start start 2024-03-29T11:43:04.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.670+0000 7f0755225640 1 librados: setting wanted keys 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.670+0000 7f0755225640 1 librados: calling monclient init 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.670+0000 7f0755225640 10 monclient: init 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 10 monclient(hunting): picked mon.noname-b con 0x7f07500a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 10 monclient(hunting): picked mon.noname-a con 0x7f075013f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 10 monclient(hunting): picked mon.noname-c con 0x7f07501457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500ac370 con 0x7f075013f650 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0750071a10 con 0x7f07500a9d70 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500ac5d0 con 0x7f07501457f0 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:04.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.671+0000 7f0755225640 10 monclient(hunting): authenticate will time out at 997.951276s 2024-03-29T11:43:04.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.675+0000 7f074ed9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f07500a9d70 0x7f075006e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:41075/0 (socket says 172.21.15.55:41075) 2024-03-29T11:43:04.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.675+0000 7f074ed9d640 1 -- 172.21.15.55:0/27778339 learned_addr learned my addr 172.21.15.55:0/27778339 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:04.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.677+0000 7f07377fe640 10 client.?.objecter ms_handle_connect 0x7f07500a9d70 2024-03-29T11:43:04.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.677+0000 7f07377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:04.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.677+0000 7f07377fe640 10 client.?.objecter ms_handle_connect 0x7f075013f650 2024-03-29T11:43:04.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.677+0000 7f07377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:04.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.678+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908867424 0 0) 0x7f07500ac370 con 0x7f075013f650 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.678+0000 7f07377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.678+0000 7f07377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.678+0000 7f07377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.678+0000 7f07377fe640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.678+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f073c004eb0 con 0x7f075013f650 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.679+0000 7f07377fe640 10 client.?.objecter ms_handle_connect 0x7f07501457f0 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.679+0000 7f07377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.679+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553226175 0 0) 0x7f0750071a10 con 0x7f07500a9d70 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.679+0000 7f07377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.679+0000 7f07377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.679+0000 7f07377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.679+0000 7f07377fe640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:43:04.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.679+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f073c0051d0 con 0x7f07500a9d70 2024-03-29T11:43:04.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.680+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624377431 0 0) 0x7f07500ac5d0 con 0x7f07501457f0 2024-03-29T11:43:04.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.680+0000 7f07377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:04.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.680+0000 7f07377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:04.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.680+0000 7f07377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:04.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.680+0000 7f07377fe640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:43:04.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.680+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0750071a10 con 0x7f07501457f0 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3073814474 0 0) 0x7f073c0051d0 con 0x7f07500a9d70 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07500ac5d0 con 0x7f07500a9d70 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0748001f30 con 0x7f07500a9d70 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient(hunting): dump: 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.686 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 607365343 0 0) 0x7f073c004eb0 con 0x7f075013f650 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f073c005cb0 con 0x7f075013f650 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0740001510 con 0x7f075013f650 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient(hunting): dump: 2024-03-29T11:43:04.687 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3826421815 0 0) 0x7f07500ac5d0 con 0x7f07500a9d70 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 >> v1:172.21.15.55:6790/0 conn(0x7f07501457f0 legacy=0x7f07501e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 >> v1:172.21.15.55:6789/0 conn(0x7f075013f650 legacy=0x7f075006e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 monclient: found mon.b 2024-03-29T11:43:04.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075006efa0 con 0x7f07500a9d70 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f07377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f0755225640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f0755225640 10 monclient: _renew_subs 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.681+0000 7f0755225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.682+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075013bdf0 con 0x7f07500a9d70 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.682+0000 7f0755225640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.682+0000 7f0755225640 10 monclient: _renew_subs 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.682+0000 7f0755225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.682+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07501336c0 con 0x7f07500a9d70 2024-03-29T11:43:04.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.682+0000 7f0755225640 1 librados: init done 2024-03-29T11:43:04.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.683+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0748004e10 con 0x7f07500a9d70 2024-03-29T11:43:04.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.683+0000 7f0755225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.683+0000 7f0755225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.683+0000 7f0755225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.683+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0750075f10 con 0x7f07500a9d70 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.684+0000 7f07377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.684+0000 7f074cd99640 4 set_mon_vals no callback set 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.685+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0748001f30 con 0x7f07500a9d70 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.685+0000 7f07377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.685+0000 7f07377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.685+0000 7f07377fe640 10 monclient: dump: 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:04.692 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.687+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f074801adf0 con 0x7f07500a9d70 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.687+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f0748001d50 con 0x7f07500a9d70 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.688+0000 7f07377fe640 10 client.4264.objecter ms_dispatch 0x7f0750000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.688+0000 7f07377fe640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.688+0000 7f07377fe640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.688+0000 7f07377fe640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T11:43:04.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.690+0000 7f074e59c640 10 monclient: get_auth_request con 0x7f073c06e890 auth_method 0 2024-03-29T11:43:04.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.691+0000 7f07377fe640 10 client.4264.objecter ms_handle_connect 0x7f073c06e890 2024-03-29T11:43:04.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.694+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0748050990 con 0x7f07500a9d70 2024-03-29T11:43:04.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.694+0000 7f07377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:04.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:04.694+0000 7f07377fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:05.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.024+0000 7f0755225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:05.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.024+0000 7f0755225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:05.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.024+0000 7f0755225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:05.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.024+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07500b3680 con 0x7f07500a9d70 2024-03-29T11:43:05.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.026+0000 7f07377fe640 1 -- 172.21.15.55:0/27778339 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f0748050990 con 0x7f07500a9d70 2024-03-29T11:43:05.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.026+0000 7f07377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:05.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.026+0000 7f07377fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:05.093 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:43:05.094 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:05.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.030+0000 7f0755225640 10 librados: watch_flush enter 2024-03-29T11:43:05.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.030+0000 7f0755225640 10 librados: watch_flush exit 2024-03-29T11:43:05.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.030+0000 7f0755225640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.030+0000 7f0755225640 10 client.4264.objecter successfully canceled tick 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.032+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 >> v1:172.21.15.171:6816/1463197163 conn(0x7f073c06e890 legacy=0x7f073c070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.032+0000 7f0755225640 10 monclient: shutdown 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.032+0000 7f0755225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.032+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 >> v1:172.21.15.171:6789/0 conn(0x7f07500a9d70 legacy=0x7f075006e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.033+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 shutdown_connections 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.033+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 >> 172.21.15.55:0/27778339 conn(0x7f0750066a00 msgr2=0x7f07500b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.033+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 shutdown_connections 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.034+0000 7f0755225640 1 -- 172.21.15.55:0/27778339 wait complete. 2024-03-29T11:43:05.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:05.034+0000 7f0755225640 1 librados: shutdown 2024-03-29T11:43:05.096 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:11.904 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:12.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.051+0000 7f35030c3640 10 monclient: get_monmap_and_config 2024-03-29T11:43:12.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.051+0000 7f35030c3640 10 monclient: build_initial_monmap 2024-03-29T11:43:12.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.051+0000 7f35030c3640 10 monclient: monmap: 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:43:12.052230+0000 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:43:12.052230+0000 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:43:12.055 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:12.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.054+0000 7f35030c3640 1 Processor -- start 2024-03-29T11:43:12.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.054+0000 7f35030c3640 1 -- start start 2024-03-29T11:43:12.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.054+0000 7f35030c3640 10 monclient: init 2024-03-29T11:43:12.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:12.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:12.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 10 monclient(hunting): picked mon.noname-c con 0x7f34f4005ff0 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:12.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 10 monclient(hunting): picked mon.noname-b con 0x7f34f4033640 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:12.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 10 monclient(hunting): picked mon.noname-a con 0x7f34f4031ee0 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:12.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f40ccd50 con 0x7f34f4031ee0 2024-03-29T11:43:12.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f4032980 con 0x7f34f4005ff0 2024-03-29T11:43:12.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f4032c60 con 0x7f34f4033640 2024-03-29T11:43:12.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:12.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.055+0000 7f35030c3640 10 monclient(hunting): authenticate will time out at 1005.335317s 2024-03-29T11:43:12.063 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.059+0000 7f35028c2640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f34f4031ee0 0x7f34f40cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:40379/0 (socket says 172.21.15.55:40379) 2024-03-29T11:43:12.063 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.059+0000 7f35028c2640 1 -- 172.21.15.55:0/3027449214 learned_addr learned my addr 172.21.15.55:0/3027449214 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:12.063 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.059+0000 7f35010bf640 1 -- 172.21.15.55:0/3027449214 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830315457 0 0) 0x7f34f40ccd50 con 0x7f34f4031ee0 2024-03-29T11:43:12.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.060+0000 7f35010bf640 10 monclient(hunting): _negotiate 2024-03-29T11:43:12.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.060+0000 7f35010bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:12.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.060+0000 7f35010bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:12.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.060+0000 7f35010bf640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:43:12.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.060+0000 7f35010bf640 1 -- 172.21.15.55:0/3027449214 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e8003520 con 0x7f34f4031ee0 2024-03-29T11:43:12.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.060+0000 7f35010bf640 1 -- 172.21.15.55:0/3027449214 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3750203482 0 0) 0x7f34e8003520 con 0x7f34f4031ee0 2024-03-29T11:43:12.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.060+0000 7f35010bf640 1 -- 172.21.15.55:0/3027449214 >> v1:172.21.15.55:6790/0 conn(0x7f34f4033640 legacy=0x7f34f40cbf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:12.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 1 -- 172.21.15.55:0/3027449214 >> v1:172.21.15.171:6789/0 conn(0x7f34f4005ff0 legacy=0x7f34f4031500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:12.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 10 monclient: _finish_hunting 0 2024-03-29T11:43:12.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 1 monclient: found mon.noname-a 2024-03-29T11:43:12.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:12.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:43:12.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 1 -- 172.21.15.55:0/3027449214 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f4032f40 con 0x7f34f4031ee0 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 10 monclient: _finish_auth 0 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 10 monclient: _check_auth_tickets 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 1 -- 172.21.15.55:0/3027449214 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f34fc04f730 con 0x7f34f4031ee0 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35010bf640 10 monclient: dump: 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:12.067 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35030c3640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.062+0000 7f35030c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.063+0000 7f35010bf640 1 -- 172.21.15.55:0/3027449214 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34fc0008d0 con 0x7f34f4031ee0 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.063+0000 7f35010bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.063+0000 7f35030c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:12.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.063+0000 7f35030c3640 4 set_mon_vals no callback set 2024-03-29T11:43:12.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.064+0000 7f35030c3640 10 monclient: shutdown 2024-03-29T11:43:12.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.064+0000 7f35030c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:12.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.064+0000 7f35030c3640 1 -- 172.21.15.55:0/3027449214 >> v1:172.21.15.55:6789/0 conn(0x7f34f4031ee0 legacy=0x7f34f40cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:12.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.064+0000 7f35030c3640 1 -- 172.21.15.55:0/3027449214 shutdown_connections 2024-03-29T11:43:12.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.064+0000 7f35030c3640 1 -- 172.21.15.55:0/3027449214 >> 172.21.15.55:0/3027449214 conn(0x7f34f40065b0 msgr2=0x7f34f4006990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:12.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.064+0000 7f35030c3640 1 -- 172.21.15.55:0/3027449214 shutdown_connections 2024-03-29T11:43:12.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.064+0000 7f35030c3640 1 -- 172.21.15.55:0/3027449214 wait complete. 2024-03-29T11:43:12.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.065+0000 7f35030c3640 10 monclient: build_initial_monmap 2024-03-29T11:43:12.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.065+0000 7f35030c3640 10 monclient: monmap: 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:12.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.066+0000 7f35030c3640 1 librados: starting msgr at 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.066+0000 7f35030c3640 1 librados: starting objecter 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.066+0000 7f35030c3640 1 Processor -- start 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.066+0000 7f35030c3640 1 -- start start 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.066+0000 7f35030c3640 1 librados: setting wanted keys 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.066+0000 7f35030c3640 1 librados: calling monclient init 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.066+0000 7f35030c3640 10 monclient: init 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 10 monclient(hunting): picked mon.noname-c con 0x7f34f4005ff0 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:12.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 10 monclient(hunting): picked mon.noname-a con 0x7f34f4031ee0 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:12.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 10 monclient(hunting): picked mon.noname-b con 0x7f34f4033640 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:12.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f416c9e0 con 0x7f34f4031ee0 2024-03-29T11:43:12.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f40a62d0 con 0x7f34f4033640 2024-03-29T11:43:12.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f40a4d50 con 0x7f34f4005ff0 2024-03-29T11:43:12.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:12.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.067+0000 7f35030c3640 10 monclient(hunting): authenticate will time out at 1005.347002s 2024-03-29T11:43:12.082 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.071+0000 7f35020c1640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f34f4005ff0 0x7f34f40a4640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:37139/0 (socket says 172.21.15.55:37139) 2024-03-29T11:43:12.082 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.071+0000 7f35020c1640 1 -- 172.21.15.55:0/3187580510 learned_addr learned my addr 172.21.15.55:0/3187580510 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.071+0000 7f34f2ffd640 10 client.?.objecter ms_handle_connect 0x7f34f4005ff0 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.071+0000 7f34f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.071+0000 7f34f2ffd640 10 client.?.objecter ms_handle_connect 0x7f34f4033640 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.071+0000 7f34f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114320342 0 0) 0x7f34f40a62d0 con 0x7f34f4033640 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e0004eb0 con 0x7f34f4033640 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2506633944 0 0) 0x7f34e0004eb0 con 0x7f34f4033640 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34f40a62d0 con 0x7f34f4033640 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f34fc04f400 con 0x7f34f4033640 2024-03-29T11:43:12.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:12.084 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1897390436 0 0) 0x7f34f40a62d0 con 0x7f34f4033640 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 >> v1:172.21.15.55:6790/0 conn(0x7f34f4005ff0 legacy=0x7f34f40a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.072+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 >> v1:172.21.15.55:6789/0 conn(0x7f34f4031ee0 legacy=0x7f34f416ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 1 monclient: found mon.b 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f40a4ee0 con 0x7f34f4033640 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f35030c3640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f35030c3640 10 monclient: _renew_subs 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f35030c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:12.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f40a1130 con 0x7f34f4033640 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f35030c3640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f35030c3640 10 monclient: _renew_subs 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f35030c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f416d760 con 0x7f34f4033640 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f35030c3640 1 librados: init done 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 10 client.4270.objecter ms_handle_connect 0x7f34f4031ee0 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.073+0000 7f34f2ffd640 10 client.4270.objecter resend_mon_ops 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.074+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34fc04fdd0 con 0x7f34f4033640 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.074+0000 7f34f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.074+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f34fc075710 con 0x7f34f4033640 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.074+0000 7f34f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.074+0000 7f34f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.074+0000 7f34f2ffd640 10 monclient: dump: 2024-03-29T11:43:12.086 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.074+0000 7f35008be640 4 set_mon_vals no callback set 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.076+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f34fc067ea0 con 0x7f34f4033640 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.076+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f34fc0ff630 con 0x7f34f4033640 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.076+0000 7f34f2ffd640 10 client.4270.objecter ms_dispatch 0x7f34fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.076+0000 7f34f2ffd640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.076+0000 7f34f2ffd640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:12.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.076+0000 7f34f2ffd640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T11:43:12.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.077+0000 7f35020c1640 10 monclient: get_auth_request con 0x7f34e006daf0 auth_method 0 2024-03-29T11:43:12.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.077+0000 7f35030c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:12.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.077+0000 7f34f2ffd640 10 client.4270.objecter ms_handle_connect 0x7f34e006daf0 2024-03-29T11:43:12.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.077+0000 7f35030c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:12.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.077+0000 7f35030c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:12.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.077+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e4005080 con 0x7f34f4033640 2024-03-29T11:43:12.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.085+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34fc0caf70 con 0x7f34f4033640 2024-03-29T11:43:12.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.085+0000 7f34f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:12.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.085+0000 7f34f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:12.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.413+0000 7f35030c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:12.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.413+0000 7f35030c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:12.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.413+0000 7f35030c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:12.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.413+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34e4005330 con 0x7f34f4033640 2024-03-29T11:43:12.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.414+0000 7f34f2ffd640 1 -- 172.21.15.55:0/3187580510 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f34fc0fece0 con 0x7f34f4033640 2024-03-29T11:43:12.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.414+0000 7f34f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:12.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.414+0000 7f34f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:12.418 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:43:12.418 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:12.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.417+0000 7f35030c3640 10 librados: watch_flush enter 2024-03-29T11:43:12.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.417+0000 7f35030c3640 10 librados: watch_flush exit 2024-03-29T11:43:12.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.417+0000 7f35030c3640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T11:43:12.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.417+0000 7f35030c3640 10 client.4270.objecter successfully canceled tick 2024-03-29T11:43:12.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.418+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 >> v1:172.21.15.171:6816/1463197163 conn(0x7f34e006daf0 legacy=0x7f34e006ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:12.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.418+0000 7f35030c3640 10 monclient: shutdown 2024-03-29T11:43:12.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.418+0000 7f35030c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:12.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.418+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 >> v1:172.21.15.171:6789/0 conn(0x7f34f4033640 legacy=0x7f34f40cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:12.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.418+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 shutdown_connections 2024-03-29T11:43:12.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.418+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 >> 172.21.15.55:0/3187580510 conn(0x7f34f40065b0 msgr2=0x7f34f402fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:12.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.418+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 shutdown_connections 2024-03-29T11:43:12.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.422+0000 7f35030c3640 1 -- 172.21.15.55:0/3187580510 wait complete. 2024-03-29T11:43:12.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:12.422+0000 7f35030c3640 1 librados: shutdown 2024-03-29T11:43:12.448 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:19.256 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:19.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.405+0000 7f02ec2fb640 10 monclient: get_monmap_and_config 2024-03-29T11:43:19.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.405+0000 7f02ec2fb640 10 monclient: build_initial_monmap 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.405+0000 7f02ec2fb640 10 monclient: monmap: 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:43:19.406318+0000 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:43:19.406318+0000 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:43:19.409 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:19.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.406+0000 7f02ec2fb640 1 Processor -- start 2024-03-29T11:43:19.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.406+0000 7f02ec2fb640 1 -- start start 2024-03-29T11:43:19.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.406+0000 7f02ec2fb640 10 monclient: init 2024-03-29T11:43:19.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.408+0000 7f02ec2fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:19.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.408+0000 7f02ec2fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:19.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ec2fb640 10 monclient(hunting): picked mon.noname-a con 0x7f02e41457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:19.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ec2fb640 10 monclient(hunting): picked mon.noname-b con 0x7f02e413f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:19.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ec2fb640 10 monclient(hunting): picked mon.noname-c con 0x7f02e40a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:19.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ec2fb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e40aaef0 con 0x7f02e41457f0 2024-03-29T11:43:19.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ec2fb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e4145bd0 con 0x7f02e40a9d70 2024-03-29T11:43:19.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ec2fb640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e4145eb0 con 0x7f02e413f650 2024-03-29T11:43:19.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ec2fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:19.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ec2fb640 10 monclient(hunting): authenticate will time out at 1012.690897s 2024-03-29T11:43:19.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ea097640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f02e41457f0 0x7f02e413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47423/0 (socket says 172.21.15.55:47423) 2024-03-29T11:43:19.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.411+0000 7f02ea097640 1 -- 172.21.15.55:0/3937398953 learned_addr learned my addr 172.21.15.55:0/3937398953 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:19.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.415+0000 7f02e9095640 1 -- 172.21.15.55:0/3937398953 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654745327 0 0) 0x7f02e40aaef0 con 0x7f02e41457f0 2024-03-29T11:43:19.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.415+0000 7f02e9095640 10 monclient(hunting): _negotiate 2024-03-29T11:43:19.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.415+0000 7f02e9095640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:19.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.415+0000 7f02e9095640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:19.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.415+0000 7f02e9095640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:43:19.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.415+0000 7f02e9095640 1 -- 172.21.15.55:0/3937398953 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d0003520 con 0x7f02e41457f0 2024-03-29T11:43:19.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 1 -- 172.21.15.55:0/3937398953 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1534841976 0 0) 0x7f02d0003520 con 0x7f02e41457f0 2024-03-29T11:43:19.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 1 -- 172.21.15.55:0/3937398953 >> v1:172.21.15.55:6790/0 conn(0x7f02e413f650 legacy=0x7f02e40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:19.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 1 -- 172.21.15.55:0/3937398953 >> v1:172.21.15.171:6789/0 conn(0x7f02e40a9d70 legacy=0x7f02e414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:19.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 10 monclient: _finish_hunting 0 2024-03-29T11:43:19.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 1 monclient: found mon.noname-a 2024-03-29T11:43:19.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:19.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:43:19.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 1 -- 172.21.15.55:0/3937398953 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4146340 con 0x7f02e41457f0 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 10 monclient: _finish_auth 0 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 10 monclient: _check_auth_tickets 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 1 -- 172.21.15.55:0/3937398953 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02e0001690 con 0x7f02e41457f0 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.416+0000 7f02e9095640 10 monclient: dump: 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:19.422 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:19.423 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:19.423 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:19.423 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:19.423 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:19.423 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:19.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.417+0000 7f02ec2fb640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:43:19.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.417+0000 7f02ec2fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:19.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.419+0000 7f02e9095640 1 -- 172.21.15.55:0/3937398953 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02e0001860 con 0x7f02e41457f0 2024-03-29T11:43:19.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.419+0000 7f02e9095640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:19.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.419+0000 7f02e9095640 1 -- 172.21.15.55:0/3937398953 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02e0001cb0 con 0x7f02e41457f0 2024-03-29T11:43:19.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.419+0000 7f02e9095640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:19.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.419+0000 7f02e9095640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:19.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.419+0000 7f02e9095640 10 monclient: dump: 2024-03-29T11:43:19.425 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:19.426 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:19.426 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:19.426 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:19.426 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:19.426 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:19.426 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:19.426 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:19.429 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:19.429 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:19.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.419+0000 7f02ec2fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:19.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.419+0000 7f02ec2fb640 4 set_mon_vals no callback set 2024-03-29T11:43:19.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.420+0000 7f02ec2fb640 10 monclient: shutdown 2024-03-29T11:43:19.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.420+0000 7f02ec2fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:19.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.420+0000 7f02ec2fb640 1 -- 172.21.15.55:0/3937398953 >> v1:172.21.15.55:6789/0 conn(0x7f02e41457f0 legacy=0x7f02e413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:19.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.420+0000 7f02ec2fb640 1 -- 172.21.15.55:0/3937398953 shutdown_connections 2024-03-29T11:43:19.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.421+0000 7f02ec2fb640 1 -- 172.21.15.55:0/3937398953 >> 172.21.15.55:0/3937398953 conn(0x7f02e4066a00 msgr2=0x7f02e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:19.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.421+0000 7f02ec2fb640 1 -- 172.21.15.55:0/3937398953 shutdown_connections 2024-03-29T11:43:19.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.421+0000 7f02ec2fb640 1 -- 172.21.15.55:0/3937398953 wait complete. 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.422+0000 7f02ec2fb640 10 monclient: build_initial_monmap 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.422+0000 7f02ec2fb640 10 monclient: monmap: 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:19.432 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.423+0000 7f02ec2fb640 1 librados: starting msgr at 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.423+0000 7f02ec2fb640 1 librados: starting objecter 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.423+0000 7f02ec2fb640 1 Processor -- start 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.425+0000 7f02ec2fb640 1 -- start start 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.425+0000 7f02ec2fb640 1 librados: setting wanted keys 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.425+0000 7f02ec2fb640 1 librados: calling monclient init 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.425+0000 7f02ec2fb640 10 monclient: init 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:19.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 10 monclient(hunting): picked mon.noname-b con 0x7f02e40a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:19.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 10 monclient(hunting): picked mon.noname-c con 0x7f02e413f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:19.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 10 monclient(hunting): picked mon.noname-a con 0x7f02e41457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:19.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e41e0350 con 0x7f02e41457f0 2024-03-29T11:43:19.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e413c880 con 0x7f02e40a9d70 2024-03-29T11:43:19.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e40acdd0 con 0x7f02e413f650 2024-03-29T11:43:19.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:19.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.426+0000 7f02ec2fb640 10 monclient(hunting): authenticate will time out at 1012.706001s 2024-03-29T11:43:19.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.429+0000 7f02e9896640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f02e413f650 0x7f02e41df530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:34535/0 (socket says 172.21.15.55:34535) 2024-03-29T11:43:19.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.429+0000 7f02e9896640 1 -- 172.21.15.55:0/4120887783 learned_addr learned my addr 172.21.15.55:0/4120887783 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.429+0000 7f02daffd640 10 client.?.objecter ms_handle_connect 0x7f02e413f650 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.430+0000 7f02daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.430+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17783446 0 0) 0x7f02e40acdd0 con 0x7f02e413f650 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.430+0000 7f02daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.430+0000 7f02daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.430+0000 7f02daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.431+0000 7f02daffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.431+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c8004eb0 con 0x7f02e413f650 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.431+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1763992703 0 0) 0x7f02c8004eb0 con 0x7f02e413f650 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.431+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02e40acdd0 con 0x7f02e413f650 2024-03-29T11:43:19.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.431+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02d40019d0 con 0x7f02e413f650 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.431+0000 7f02daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.431+0000 7f02daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.432+0000 7f02daffd640 10 monclient(hunting): dump: 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:19.440 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.432+0000 7f02daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.432+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3559832279 0 0) 0x7f02e40acdd0 con 0x7f02e413f650 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.432+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 >> v1:172.21.15.171:6789/0 conn(0x7f02e40a9d70 legacy=0x7f02e40ad260 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.432+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 >> v1:172.21.15.55:6789/0 conn(0x7f02e41457f0 legacy=0x7f02e41dfc40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.433+0000 7f02daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.433+0000 7f02daffd640 1 monclient: found mon.c 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.433+0000 7f02daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.433+0000 7f02daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.433+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4071690 con 0x7f02e413f650 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.433+0000 7f02daffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.433+0000 7f02daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:19.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.433+0000 7f02daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02d4001cd0 con 0x7f02e413f650 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f02d4003a60 con 0x7f02e413f650 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02daffd640 10 monclient: dump: 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:19.442 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02ec2fb640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02dbfff640 4 set_mon_vals no callback set 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02ec2fb640 10 monclient: _renew_subs 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02ec2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e40ad970 con 0x7f02e413f650 2024-03-29T11:43:19.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02ec2fb640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:19.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02ec2fb640 10 monclient: _renew_subs 2024-03-29T11:43:19.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02ec2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:19.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.434+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e4071e30 con 0x7f02e413f650 2024-03-29T11:43:19.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.435+0000 7f02ec2fb640 1 librados: init done 2024-03-29T11:43:19.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.438+0000 7f02ec2fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:19.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.438+0000 7f02ec2fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:19.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.438+0000 7f02ec2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:19.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.438+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e40acc40 con 0x7f02e413f650 2024-03-29T11:43:19.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.439+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f02d4003bc0 con 0x7f02e413f650 2024-03-29T11:43:19.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.440+0000 7f02ea097640 10 monclient: get_auth_request con 0x7f02c806dc90 auth_method 0 2024-03-29T11:43:19.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.440+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f02d4084b70 con 0x7f02e413f650 2024-03-29T11:43:19.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.440+0000 7f02daffd640 10 client.4274.objecter ms_dispatch 0x7f02e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:19.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.440+0000 7f02daffd640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:19.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.440+0000 7f02daffd640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:19.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.440+0000 7f02daffd640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T11:43:19.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.440+0000 7f02daffd640 10 client.4274.objecter ms_handle_connect 0x7f02c806dc90 2024-03-29T11:43:19.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.449+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02d4050280 con 0x7f02e413f650 2024-03-29T11:43:19.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.449+0000 7f02daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:19.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.449+0000 7f02daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:19.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.771+0000 7f02ec2fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:19.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.771+0000 7f02ec2fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:19.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.771+0000 7f02ec2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:19.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.771+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02e4071500 con 0x7f02e413f650 2024-03-29T11:43:19.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.773+0000 7f02daffd640 1 -- 172.21.15.55:0/4120887783 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f02d4052380 con 0x7f02e413f650 2024-03-29T11:43:19.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.773+0000 7f02daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:19.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.773+0000 7f02daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:19.777 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:43:19.777 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:19.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 10 librados: watch_flush enter 2024-03-29T11:43:19.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 10 librados: watch_flush exit 2024-03-29T11:43:19.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T11:43:19.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 10 client.4274.objecter successfully canceled tick 2024-03-29T11:43:19.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 >> v1:172.21.15.171:6816/1463197163 conn(0x7f02c806dc90 legacy=0x7f02c8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:19.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 10 monclient: shutdown 2024-03-29T11:43:19.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:19.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 >> v1:172.21.15.55:6790/0 conn(0x7f02e413f650 legacy=0x7f02e41df530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:19.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 shutdown_connections 2024-03-29T11:43:19.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.777+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 >> 172.21.15.55:0/4120887783 conn(0x7f02e4066a00 msgr2=0x7f02e4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:19.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.779+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 shutdown_connections 2024-03-29T11:43:19.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.779+0000 7f02ec2fb640 1 -- 172.21.15.55:0/4120887783 wait complete. 2024-03-29T11:43:19.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:19.779+0000 7f02ec2fb640 1 librados: shutdown 2024-03-29T11:43:19.800 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:26.607 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:26.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.770+0000 7fded25cf640 10 monclient: get_monmap_and_config 2024-03-29T11:43:26.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.770+0000 7fded25cf640 10 monclient: build_initial_monmap 2024-03-29T11:43:26.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.770+0000 7fded25cf640 10 monclient: monmap: 2024-03-29T11:43:26.774 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:26.774 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:26.774 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:43:26.771838+0000 2024-03-29T11:43:26.774 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:43:26.771838+0000 2024-03-29T11:43:26.775 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:26.775 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:26.775 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:26.775 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:43:26.775 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:43:26.775 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:26.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.771+0000 7fded25cf640 1 Processor -- start 2024-03-29T11:43:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.772+0000 7fded25cf640 1 -- start start 2024-03-29T11:43:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.772+0000 7fded25cf640 10 monclient: init 2024-03-29T11:43:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fded25cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fded25cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fded25cf640 10 monclient(hunting): picked mon.noname-c con 0x7fdecc1457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:26.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fded25cf640 10 monclient(hunting): picked mon.noname-b con 0x7fdecc13f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:26.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fded25cf640 10 monclient(hunting): picked mon.noname-a con 0x7fdecc0a9d70 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:26.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fded0b6c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fdecc0a9d70 0x7fdecc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:57460/0 (socket says 172.21.15.55:57460) 2024-03-29T11:43:26.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fdecbfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdecc1457f0 0x7fdecc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:51964/0 (socket says 172.21.15.55:51964) 2024-03-29T11:43:26.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fded0b6c640 1 -- 172.21.15.55:0/2026941744 learned_addr learned my addr 172.21.15.55:0/2026941744 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:26.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fded25cf640 1 -- 172.21.15.55:0/2026941744 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdecc0aaef0 con 0x7fdecc0a9d70 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.773+0000 7fded25cf640 1 -- 172.21.15.55:0/2026941744 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdecc145bd0 con 0x7fdecc1457f0 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fded25cf640 1 -- 172.21.15.55:0/2026941744 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdecc145eb0 con 0x7fdecc13f650 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fded25cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fded25cf640 10 monclient(hunting): authenticate will time out at 1020.053719s 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 1 -- 172.21.15.55:0/2026941744 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908740078 0 0) 0x7fdecc145bd0 con 0x7fdecc1457f0 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 1 -- 172.21.15.55:0/2026941744 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb4003520 con 0x7fdecc1457f0 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 1 -- 172.21.15.55:0/2026941744 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1360108356 0 0) 0x7fdeb4003520 con 0x7fdecc1457f0 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 1 -- 172.21.15.55:0/2026941744 >> v1:172.21.15.55:6790/0 conn(0x7fdecc13f650 legacy=0x7fdecc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 1 -- 172.21.15.55:0/2026941744 >> v1:172.21.15.55:6789/0 conn(0x7fdecc0a9d70 legacy=0x7fdecc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:26.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 1 monclient: found mon.noname-c 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 1 -- 172.21.15.55:0/2026941744 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc146340 con 0x7fdecc1457f0 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 1 -- 172.21.15.55:0/2026941744 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdebc001690 con 0x7fdecc1457f0 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fdecaffd640 10 monclient: dump: 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.783 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fded25cf640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.774+0000 7fded25cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.775+0000 7fdecaffd640 1 -- 172.21.15.55:0/2026941744 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdebc001860 con 0x7fdecc1457f0 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.775+0000 7fdecaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.775+0000 7fdecaffd640 1 -- 172.21.15.55:0/2026941744 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdebc001c70 con 0x7fdecc1457f0 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.775+0000 7fdecaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.775+0000 7fdecaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.775+0000 7fdecaffd640 10 monclient: dump: 2024-03-29T11:43:26.784 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.775+0000 7fded25cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.775+0000 7fded25cf640 4 set_mon_vals no callback set 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.776+0000 7fded25cf640 10 monclient: shutdown 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.776+0000 7fded25cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.776+0000 7fded25cf640 1 -- 172.21.15.55:0/2026941744 >> v1:172.21.15.171:6789/0 conn(0x7fdecc1457f0 legacy=0x7fdecc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.777+0000 7fded25cf640 1 -- 172.21.15.55:0/2026941744 shutdown_connections 2024-03-29T11:43:26.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.777+0000 7fded25cf640 1 -- 172.21.15.55:0/2026941744 >> 172.21.15.55:0/2026941744 conn(0x7fdecc066a00 msgr2=0x7fdecc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:26.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.778+0000 7fded25cf640 1 -- 172.21.15.55:0/2026941744 shutdown_connections 2024-03-29T11:43:26.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.778+0000 7fded25cf640 1 -- 172.21.15.55:0/2026941744 wait complete. 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.779+0000 7fded25cf640 10 monclient: build_initial_monmap 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.779+0000 7fded25cf640 10 monclient: monmap: 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:43:26.791 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.779+0000 7fded25cf640 1 librados: starting msgr at 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.779+0000 7fded25cf640 1 librados: starting objecter 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.779+0000 7fded25cf640 1 Processor -- start 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.780+0000 7fded25cf640 1 -- start start 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.780+0000 7fded25cf640 1 librados: setting wanted keys 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.780+0000 7fded25cf640 1 librados: calling monclient init 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.780+0000 7fded25cf640 10 monclient: init 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 10 monclient(hunting): picked mon.noname-a con 0x7fdecc0a9d70 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 10 monclient(hunting): picked mon.noname-c con 0x7fdecc13f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 10 monclient(hunting): picked mon.noname-b con 0x7fdecc1457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdecc1d8150 con 0x7fdecc0a9d70 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdecc0af4c0 con 0x7fdecc1457f0 2024-03-29T11:43:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdecc1d7e00 con 0x7fdecc13f650 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded25cf640 10 monclient(hunting): authenticate will time out at 1020.061251s 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded0b6c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdecc1457f0 0x7fdecc0ab970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:52258/0 (socket says 172.21.15.55:52258) 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.781+0000 7fded0b6c640 1 -- 172.21.15.55:0/2650046392 learned_addr learned my addr 172.21.15.55:0/2650046392 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 client.?.objecter ms_handle_connect 0x7fdecc1457f0 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 client.?.objecter ms_handle_connect 0x7fdecc13f650 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 client.?.objecter ms_handle_connect 0x7fdecc0a9d70 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650246863 0 0) 0x7fdecc1d7e00 con 0x7fdecc13f650 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb8004eb0 con 0x7fdecc13f650 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.782+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561210666 0 0) 0x7fdecc0af4c0 con 0x7fdecc1457f0 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdecc1d7e00 con 0x7fdecc1457f0 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 206965530 0 0) 0x7fdeb8004eb0 con 0x7fdecc13f650 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdecc0af4c0 con 0x7fdecc13f650 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdec0001510 con 0x7fdecc13f650 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:26.794 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2056544568 0 0) 0x7fdecc1d7e00 con 0x7fdecc1457f0 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdeb8004eb0 con 0x7fdecc1457f0 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdec4001bb0 con 0x7fdecc1457f0 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3111843834 0 0) 0x7fdecc0af4c0 con 0x7fdecc13f650 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 >> v1:172.21.15.171:6789/0 conn(0x7fdecc1457f0 legacy=0x7fdecc0ab970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.783+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 >> v1:172.21.15.55:6789/0 conn(0x7fdecc0a9d70 legacy=0x7fdecc1d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 1 monclient: found mon.c 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc13a570 con 0x7fdecc13f650 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdec0001670 con 0x7fdecc13f650 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdec0001d00 con 0x7fdecc13f650 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdec8ff9640 10 monclient: dump: 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:26.797 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fded25cf640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fded25cf640 10 monclient: _renew_subs 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fded25cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdecc13a820 con 0x7fdecc13f650 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fded25cf640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fded25cf640 10 monclient: _renew_subs 2024-03-29T11:43:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fded25cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdecc142390 con 0x7fdecc13f650 2024-03-29T11:43:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fded25cf640 1 librados: init done 2024-03-29T11:43:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.784+0000 7fdeca7fc640 4 set_mon_vals no callback set 2024-03-29T11:43:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.787+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fdec0019780 con 0x7fdecc13f650 2024-03-29T11:43:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.787+0000 7fded25cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.787+0000 7fded25cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.787+0000 7fded25cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.787+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeac005080 con 0x7fdecc13f650 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.788+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fdec0083740 con 0x7fdecc13f650 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.788+0000 7fdec8ff9640 10 client.4277.objecter ms_dispatch 0x7fdecc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.788+0000 7fdec8ff9640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.788+0000 7fdec8ff9640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.788+0000 7fdec8ff9640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.789+0000 7fdecbfff640 10 monclient: get_auth_request con 0x7fdeb806df60 auth_method 0 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.789+0000 7fdec8ff9640 10 client.4277.objecter ms_handle_connect 0x7fdeb806df60 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.796+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdec004f020 con 0x7fdecc13f650 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.796+0000 7fdec8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:26.796+0000 7fdec8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:27.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.170+0000 7fded25cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:27.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.170+0000 7fded25cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:27.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.170+0000 7fded25cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:27.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.170+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeac005330 con 0x7fdecc13f650 2024-03-29T11:43:27.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.171+0000 7fdec8ff9640 1 -- 172.21.15.55:0/2650046392 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fdec0050f50 con 0x7fdecc13f650 2024-03-29T11:43:27.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.172+0000 7fdec8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:27.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.172+0000 7fdec8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:27.175 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:43:27.175 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:27.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 10 librados: watch_flush enter 2024-03-29T11:43:27.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 10 librados: watch_flush exit 2024-03-29T11:43:27.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-29T11:43:27.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 10 client.4277.objecter successfully canceled tick 2024-03-29T11:43:27.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 >> v1:172.21.15.171:6816/1463197163 conn(0x7fdeb806df60 legacy=0x7fdeb8070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:27.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 10 monclient: shutdown 2024-03-29T11:43:27.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:27.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 >> v1:172.21.15.55:6790/0 conn(0x7fdecc13f650 legacy=0x7fdecc139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:27.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 shutdown_connections 2024-03-29T11:43:27.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.175+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 >> 172.21.15.55:0/2650046392 conn(0x7fdecc066a00 msgr2=0x7fdecc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:27.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.176+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 shutdown_connections 2024-03-29T11:43:27.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.176+0000 7fded25cf640 1 -- 172.21.15.55:0/2650046392 wait complete. 2024-03-29T11:43:27.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:27.176+0000 7fded25cf640 1 librados: shutdown 2024-03-29T11:43:27.195 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:34.002 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:34.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.132+0000 7f61feb56640 10 monclient: get_monmap_and_config 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.132+0000 7f61feb56640 10 monclient: build_initial_monmap 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.132+0000 7f61feb56640 10 monclient: monmap: 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:43:34.133275+0000 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:43:34.133275+0000 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:43:34.136 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:34.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.133+0000 7f61feb56640 1 Processor -- start 2024-03-29T11:43:34.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.133+0000 7f61feb56640 1 -- start start 2024-03-29T11:43:34.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.133+0000 7f61feb56640 10 monclient: init 2024-03-29T11:43:34.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:34.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:34.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 10 monclient(hunting): picked mon.noname-b con 0x7f61f8071c30 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:34.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 10 monclient(hunting): picked mon.noname-a con 0x7f61f80a9750 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:34.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 10 monclient(hunting): picked mon.noname-c con 0x7f61f813e1e0 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:34.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80ab300 con 0x7f61f80a9750 2024-03-29T11:43:34.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80ac2f0 con 0x7f61f813e1e0 2024-03-29T11:43:34.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80ad8d0 con 0x7f61f8071c30 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61feb56640 10 monclient(hunting): authenticate will time out at 1027.414182s 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61fd0f3640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f61f813e1e0 0x7f61f80ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:59238/0 (socket says 172.21.15.55:59238) 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.134+0000 7f61fd0f3640 1 -- 172.21.15.55:0/4261443930 learned_addr learned my addr 172.21.15.55:0/4261443930 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.135+0000 7f61f77fe640 1 -- 172.21.15.55:0/4261443930 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413724953 0 0) 0x7f61f80ac2f0 con 0x7f61f813e1e0 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.135+0000 7f61f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.135+0000 7f61f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.135+0000 7f61f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.135+0000 7f61f77fe640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:43:34.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.135+0000 7f61f77fe640 1 -- 172.21.15.55:0/4261443930 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e8003520 con 0x7f61f813e1e0 2024-03-29T11:43:34.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.135+0000 7f61f77fe640 1 -- 172.21.15.55:0/4261443930 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3213202239 0 0) 0x7f61e8003520 con 0x7f61f813e1e0 2024-03-29T11:43:34.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.135+0000 7f61f77fe640 1 -- 172.21.15.55:0/4261443930 >> v1:172.21.15.55:6790/0 conn(0x7f61f8071c30 legacy=0x7f61f80b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:34.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.135+0000 7f61f77fe640 1 -- 172.21.15.55:0/4261443930 >> v1:172.21.15.55:6789/0 conn(0x7f61f80a9750 legacy=0x7f61f813dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 1 monclient: found mon.noname-c 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 1 -- 172.21.15.55:0/4261443930 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f80ada60 con 0x7f61f813e1e0 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 1 -- 172.21.15.55:0/4261443930 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f61f0001a10 con 0x7f61f813e1e0 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:34.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61f77fe640 10 monclient: dump: 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61feb56640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:43:34.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.137+0000 7f61feb56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61f77fe640 1 -- 172.21.15.55:0/4261443930 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61f0001be0 con 0x7f61f813e1e0 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61f77fe640 1 -- 172.21.15.55:0/4261443930 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f61f0002030 con 0x7f61f813e1e0 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61f77fe640 10 monclient: dump: 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:34.144 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61feb56640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61feb56640 4 set_mon_vals no callback set 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61feb56640 10 monclient: shutdown 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61feb56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.138+0000 7f61feb56640 1 -- 172.21.15.55:0/4261443930 >> v1:172.21.15.171:6789/0 conn(0x7f61f813e1e0 legacy=0x7f61f80ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.139+0000 7f61feb56640 1 -- 172.21.15.55:0/4261443930 shutdown_connections 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.139+0000 7f61feb56640 1 -- 172.21.15.55:0/4261443930 >> 172.21.15.55:0/4261443930 conn(0x7f61f806b660 msgr2=0x7f61f806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.139+0000 7f61feb56640 1 -- 172.21.15.55:0/4261443930 shutdown_connections 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.139+0000 7f61feb56640 1 -- 172.21.15.55:0/4261443930 wait complete. 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.140+0000 7f61feb56640 10 monclient: build_initial_monmap 2024-03-29T11:43:34.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.140+0000 7f61feb56640 10 monclient: monmap: 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.140+0000 7f61feb56640 1 librados: starting msgr at 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.140+0000 7f61feb56640 1 librados: starting objecter 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.140+0000 7f61feb56640 1 Processor -- start 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.140+0000 7f61feb56640 1 -- start start 2024-03-29T11:43:34.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.140+0000 7f61feb56640 1 librados: setting wanted keys 2024-03-29T11:43:34.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.140+0000 7f61feb56640 1 librados: calling monclient init 2024-03-29T11:43:34.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.140+0000 7f61feb56640 10 monclient: init 2024-03-29T11:43:34.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:34.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:34.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 10 monclient(hunting): picked mon.noname-c con 0x7f61f8071c30 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:34.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 10 monclient(hunting): picked mon.noname-b con 0x7f61f80a9750 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:34.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61fc8f2640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f61f8071c30 0x7f61f81dbce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:35738/0 (socket says 172.21.15.55:35738) 2024-03-29T11:43:34.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 10 monclient(hunting): picked mon.noname-a con 0x7f61f813e1e0 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f806fee0 con 0x7f61f813e1e0 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f81db7c0 con 0x7f61f80a9750 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f81dc3f0 con 0x7f61f8071c30 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61feb56640 10 monclient(hunting): authenticate will time out at 1027.421238s 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61fc8f2640 1 -- 172.21.15.55:0/3552009833 learned_addr learned my addr 172.21.15.55:0/3552009833 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61f57fa640 10 client.?.objecter ms_handle_connect 0x7f61f8071c30 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61f57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941082629 0 0) 0x7f61f81dc3f0 con 0x7f61f8071c30 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61f57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61f57fa640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:43:34.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.141+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61dc004eb0 con 0x7f61f8071c30 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 225237040 0 0) 0x7f61dc004eb0 con 0x7f61f8071c30 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f81dc3f0 con 0x7f61f8071c30 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f61ec001a30 con 0x7f61f8071c30 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 10 monclient(hunting): dump: 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:34.151 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3671063754 0 0) 0x7f61f81dc3f0 con 0x7f61f8071c30 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 >> v1:172.21.15.171:6789/0 conn(0x7f61f80a9750 legacy=0x7f61f8070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.142+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 >> v1:172.21.15.55:6789/0 conn(0x7f61f813e1e0 legacy=0x7f61f806e3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 1 monclient: found mon.c 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f81dc580 con 0x7f61f8071c30 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61feb56640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:43:34.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61feb56640 10 monclient: _renew_subs 2024-03-29T11:43:34.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61feb56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:34.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f80705e0 con 0x7f61f8071c30 2024-03-29T11:43:34.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61feb56640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:34.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61feb56640 10 monclient: _renew_subs 2024-03-29T11:43:34.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61feb56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:34.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61ec001d30 con 0x7f61f8071c30 2024-03-29T11:43:34.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f80715a0 con 0x7f61f8071c30 2024-03-29T11:43:34.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:34.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f61ec001a30 con 0x7f61f8071c30 2024-03-29T11:43:34.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:34.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:34.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f57fa640 10 monclient: dump: 2024-03-29T11:43:34.154 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.145+0000 7f61f6ffd640 4 set_mon_vals no callback set 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.146+0000 7f61feb56640 1 librados: init done 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.147+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f61ec003980 con 0x7f61f8071c30 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.147+0000 7f61feb56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.147+0000 7f61feb56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:34.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.147+0000 7f61feb56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:34.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.147+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f80708e0 con 0x7f61f8071c30 2024-03-29T11:43:34.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.148+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f61ec054830 con 0x7f61f8071c30 2024-03-29T11:43:34.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.148+0000 7f61f57fa640 10 client.4283.objecter ms_dispatch 0x7f61f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:34.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.148+0000 7f61f57fa640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:34.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.148+0000 7f61f57fa640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:34.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.148+0000 7f61f57fa640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T11:43:34.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.149+0000 7f61f7fff640 10 monclient: get_auth_request con 0x7f61dc06dd10 auth_method 0 2024-03-29T11:43:34.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.149+0000 7f61f57fa640 10 client.4283.objecter ms_handle_connect 0x7f61dc06dd10 2024-03-29T11:43:34.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.156+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61ec050ad0 con 0x7f61f8071c30 2024-03-29T11:43:34.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.156+0000 7f61f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:34.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.156+0000 7f61f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:34.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.481+0000 7f61feb56640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:34.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.481+0000 7f61feb56640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:34.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.481+0000 7f61feb56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:34.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.481+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61f80708e0 con 0x7f61f8071c30 2024-03-29T11:43:34.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.481+0000 7f61f57fa640 1 -- 172.21.15.55:0/3552009833 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f61f80708e0 con 0x7f61f8071c30 2024-03-29T11:43:34.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.481+0000 7f61f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:34.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.481+0000 7f61f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:34.486 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:43:34.486 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:34.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.484+0000 7f61feb56640 10 librados: watch_flush enter 2024-03-29T11:43:34.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.484+0000 7f61feb56640 10 librados: watch_flush exit 2024-03-29T11:43:34.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.484+0000 7f61feb56640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T11:43:34.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.484+0000 7f61feb56640 10 client.4283.objecter successfully canceled tick 2024-03-29T11:43:34.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.484+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 >> v1:172.21.15.171:6816/1463197163 conn(0x7f61dc06dd10 legacy=0x7f61dc0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:34.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.484+0000 7f61feb56640 10 monclient: shutdown 2024-03-29T11:43:34.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.485+0000 7f61feb56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:34.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.485+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 >> v1:172.21.15.55:6790/0 conn(0x7f61f8071c30 legacy=0x7f61f81dbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:34.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.485+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 shutdown_connections 2024-03-29T11:43:34.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.485+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 >> 172.21.15.55:0/3552009833 conn(0x7f61f806b660 msgr2=0x7f61f806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:34.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.485+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 shutdown_connections 2024-03-29T11:43:34.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.485+0000 7f61feb56640 1 -- 172.21.15.55:0/3552009833 wait complete. 2024-03-29T11:43:34.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:34.485+0000 7f61feb56640 1 librados: shutdown 2024-03-29T11:43:34.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:41.310 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:41.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.456+0000 7f93167a8640 10 monclient: get_monmap_and_config 2024-03-29T11:43:41.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.456+0000 7f93167a8640 10 monclient: build_initial_monmap 2024-03-29T11:43:41.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.456+0000 7f93167a8640 10 monclient: monmap: 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:43:41.458090+0000 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:43:41.458090+0000 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:43:41.461 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:41.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.458+0000 7f93167a8640 1 Processor -- start 2024-03-29T11:43:41.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.458+0000 7f93167a8640 1 -- start start 2024-03-29T11:43:41.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.458+0000 7f93167a8640 10 monclient: init 2024-03-29T11:43:41.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:41.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:41.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 10 monclient(hunting): picked mon.noname-c con 0x7f93101457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:41.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 10 monclient(hunting): picked mon.noname-b con 0x7f931013f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:41.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 10 monclient(hunting): picked mon.noname-a con 0x7f93100a9d70 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:41.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93100aaef0 con 0x7f93100a9d70 2024-03-29T11:43:41.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9310145bd0 con 0x7f93101457f0 2024-03-29T11:43:41.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9310145eb0 con 0x7f931013f650 2024-03-29T11:43:41.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:41.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.459+0000 7f93167a8640 10 monclient(hunting): authenticate will time out at 1034.739404s 2024-03-29T11:43:41.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.460+0000 7f930ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f93101457f0 0x7f931013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:50252/0 (socket says 172.21.15.55:50252) 2024-03-29T11:43:41.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.460+0000 7f930ffff640 1 -- 172.21.15.55:0/278430748 learned_addr learned my addr 172.21.15.55:0/278430748 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:41.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.461+0000 7f930effd640 1 -- 172.21.15.55:0/278430748 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049231570 0 0) 0x7f9310145bd0 con 0x7f93101457f0 2024-03-29T11:43:41.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.461+0000 7f930effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:41.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.461+0000 7f930effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:41.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.461+0000 7f930effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:41.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.461+0000 7f930effd640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:43:41.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.461+0000 7f930effd640 1 -- 172.21.15.55:0/278430748 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92fc003520 con 0x7f93101457f0 2024-03-29T11:43:41.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 1 -- 172.21.15.55:0/278430748 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1100877602 0 0) 0x7f92fc003520 con 0x7f93101457f0 2024-03-29T11:43:41.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 1 -- 172.21.15.55:0/278430748 >> v1:172.21.15.55:6790/0 conn(0x7f931013f650 legacy=0x7f93100b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:41.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 1 -- 172.21.15.55:0/278430748 >> v1:172.21.15.55:6789/0 conn(0x7f93100a9d70 legacy=0x7f931014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:41.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:41.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 1 monclient: found mon.noname-c 2024-03-29T11:43:41.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:41.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 1 -- 172.21.15.55:0/278430748 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9310146340 con 0x7f93101457f0 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 10 monclient: _finish_auth 0 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 1 -- 172.21.15.55:0/278430748 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9308001a10 con 0x7f93101457f0 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f930effd640 10 monclient: dump: 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:41.468 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f93167a8640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.463+0000 7f93167a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.465+0000 7f930effd640 1 -- 172.21.15.55:0/278430748 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9308001be0 con 0x7f93101457f0 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.465+0000 7f930effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.465+0000 7f930effd640 1 -- 172.21.15.55:0/278430748 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9308002030 con 0x7f93101457f0 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.465+0000 7f930effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.465+0000 7f930effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:41.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.465+0000 7f930effd640 10 monclient: dump: 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.465+0000 7f93167a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.465+0000 7f93167a8640 4 set_mon_vals no callback set 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.466+0000 7f93167a8640 10 monclient: shutdown 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.466+0000 7f93167a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:41.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.466+0000 7f93167a8640 1 -- 172.21.15.55:0/278430748 >> v1:172.21.15.171:6789/0 conn(0x7f93101457f0 legacy=0x7f931013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:41.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.468+0000 7f93167a8640 1 -- 172.21.15.55:0/278430748 shutdown_connections 2024-03-29T11:43:41.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.468+0000 7f93167a8640 1 -- 172.21.15.55:0/278430748 >> 172.21.15.55:0/278430748 conn(0x7f9310066a00 msgr2=0x7f9310066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:41.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.468+0000 7f93167a8640 1 -- 172.21.15.55:0/278430748 shutdown_connections 2024-03-29T11:43:41.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.468+0000 7f93167a8640 1 -- 172.21.15.55:0/278430748 wait complete. 2024-03-29T11:43:41.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.469+0000 7f93167a8640 10 monclient: build_initial_monmap 2024-03-29T11:43:41.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.469+0000 7f93167a8640 10 monclient: monmap: 2024-03-29T11:43:41.473 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:41.473 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:41.473 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:43:41.473 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:43:41.473 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:41.473 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:41.474 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:41.474 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:43:41.474 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:43:41.474 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:41.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.469+0000 7f93167a8640 1 librados: starting msgr at 2024-03-29T11:43:41.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.469+0000 7f93167a8640 1 librados: starting objecter 2024-03-29T11:43:41.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.470+0000 7f93167a8640 1 Processor -- start 2024-03-29T11:43:41.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.470+0000 7f93167a8640 1 -- start start 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.470+0000 7f93167a8640 1 librados: setting wanted keys 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.470+0000 7f93167a8640 1 librados: calling monclient init 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.470+0000 7f93167a8640 10 monclient: init 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.471+0000 7f93167a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.471+0000 7f93167a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.471+0000 7f93167a8640 10 monclient(hunting): picked mon.noname-c con 0x7f93100a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.471+0000 7f93167a8640 10 monclient(hunting): picked mon.noname-b con 0x7f931013f650 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.471+0000 7f93167a8640 10 monclient(hunting): picked mon.noname-a con 0x7f93101457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.471+0000 7f93167a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93101426c0 con 0x7f93101457f0 2024-03-29T11:43:41.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.471+0000 7f93167a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9310144750 con 0x7f931013f650 2024-03-29T11:43:41.476 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.471+0000 7f93167a8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9310142130 con 0x7f93100a9d70 2024-03-29T11:43:41.476 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.472+0000 7f93167a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:41.476 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.472+0000 7f93167a8640 10 monclient(hunting): authenticate will time out at 1034.751663s 2024-03-29T11:43:41.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.473+0000 7f930f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f931013f650 0x7f931013cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:50436/0 (socket says 172.21.15.55:50436) 2024-03-29T11:43:41.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.473+0000 7f930f7fe640 1 -- 172.21.15.55:0/2783007346 learned_addr learned my addr 172.21.15.55:0/2783007346 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:41.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.474+0000 7f930cff9640 10 client.?.objecter ms_handle_connect 0x7f931013f650 2024-03-29T11:43:41.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.474+0000 7f930cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:41.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776819290 0 0) 0x7f9310144750 con 0x7f931013f650 2024-03-29T11:43:41.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:41.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:41.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:41.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:43:41.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0004eb0 con 0x7f931013f650 2024-03-29T11:43:41.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 client.?.objecter ms_handle_connect 0x7f93101457f0 2024-03-29T11:43:41.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 731978690 0 0) 0x7f92f0004eb0 con 0x7f931013f650 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9310144750 con 0x7f931013f650 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9300001960 con 0x7f931013f650 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): dump: 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:41.481 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130473785 0 0) 0x7f93101426c0 con 0x7f93101457f0 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:43:41.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.476+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0004eb0 con 0x7f93101457f0 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 client.?.objecter ms_handle_connect 0x7f93100a9d70 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650892247 0 0) 0x7f9310142130 con 0x7f93100a9d70 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0005890 con 0x7f93100a9d70 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 957824262 0 0) 0x7f9310144750 con 0x7f931013f650 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 >> v1:172.21.15.55:6790/0 conn(0x7f93100a9d70 legacy=0x7f9310144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 >> v1:172.21.15.55:6789/0 conn(0x7f93101457f0 legacy=0x7f93101ee8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:41.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 1 monclient: found mon.b 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931013fda0 con 0x7f931013f650 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f930cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f93167a8640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f93167a8640 10 monclient: _renew_subs 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f93167a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931013fff0 con 0x7f931013f650 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f93167a8640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f93167a8640 10 monclient: _renew_subs 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f93167a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:41.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9310144ba0 con 0x7f931013f650 2024-03-29T11:43:41.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.480+0000 7f93167a8640 1 librados: init done 2024-03-29T11:43:41.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f93167a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:41.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f93167a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f93167a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9300001c40 con 0x7f931013f650 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9310075f10 con 0x7f931013f650 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f930cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9300003a40 con 0x7f931013f650 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f930cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f930cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f930cff9640 10 monclient: dump: 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:41.487 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:41.488 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:41.488 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:41.488 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:41.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.482+0000 7f930e7fc640 4 set_mon_vals no callback set 2024-03-29T11:43:41.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.484+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f9300003ba0 con 0x7f931013f650 2024-03-29T11:43:41.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.485+0000 7f930ffff640 10 monclient: get_auth_request con 0x7f92f006dfe0 auth_method 0 2024-03-29T11:43:41.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.485+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f93000547a0 con 0x7f931013f650 2024-03-29T11:43:41.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.485+0000 7f930cff9640 10 client.4285.objecter ms_dispatch 0x7f9310000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:41.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.485+0000 7f930cff9640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:41.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.485+0000 7f930cff9640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:41.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.485+0000 7f930cff9640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-29T11:43:41.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.485+0000 7f930cff9640 10 client.4285.objecter ms_handle_connect 0x7f92f006dfe0 2024-03-29T11:43:41.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.490+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9300050a40 con 0x7f931013f650 2024-03-29T11:43:41.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.490+0000 7f930cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:41.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.490+0000 7f930cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:41.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.829+0000 7f93167a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:41.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.829+0000 7f93167a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:41.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.829+0000 7f93167a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:41.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.829+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9310074f50 con 0x7f931013f650 2024-03-29T11:43:41.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.831+0000 7f930cff9640 1 -- 172.21.15.55:0/2783007346 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f9300050a40 con 0x7f931013f650 2024-03-29T11:43:41.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.832+0000 7f930cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:41.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.832+0000 7f930cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:41.836 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:43:41.836 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:41.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 10 librados: watch_flush enter 2024-03-29T11:43:41.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 10 librados: watch_flush exit 2024-03-29T11:43:41.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-29T11:43:41.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 10 client.4285.objecter successfully canceled tick 2024-03-29T11:43:41.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 >> v1:172.21.15.171:6816/1463197163 conn(0x7f92f006dfe0 legacy=0x7f92f00704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:41.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 10 monclient: shutdown 2024-03-29T11:43:41.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:41.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 >> v1:172.21.15.171:6789/0 conn(0x7f931013f650 legacy=0x7f931013cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:41.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 shutdown_connections 2024-03-29T11:43:41.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.835+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 >> 172.21.15.55:0/2783007346 conn(0x7f9310066a00 msgr2=0x7f931013b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:41.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.836+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 shutdown_connections 2024-03-29T11:43:41.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.836+0000 7f93167a8640 1 -- 172.21.15.55:0/2783007346 wait complete. 2024-03-29T11:43:41.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:41.836+0000 7f93167a8640 1 librados: shutdown 2024-03-29T11:43:41.857 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:48.665 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:48.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.804+0000 7f492b855640 10 monclient: get_monmap_and_config 2024-03-29T11:43:48.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.804+0000 7f492b855640 10 monclient: build_initial_monmap 2024-03-29T11:43:48.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.804+0000 7f492b855640 10 monclient: monmap: 2024-03-29T11:43:48.819 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:48.819 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:48.819 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:43:48.806087+0000 2024-03-29T11:43:48.819 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:43:48.806087+0000 2024-03-29T11:43:48.819 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:48.819 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.805+0000 7f492b855640 1 Processor -- start 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.806+0000 7f492b855640 1 -- start start 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.806+0000 7f492b855640 10 monclient: init 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 10 monclient(hunting): picked mon.noname-b con 0x7f49240a9930 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 10 monclient(hunting): picked mon.noname-c con 0x7f4924149dc0 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:48.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 10 monclient(hunting): picked mon.noname-a con 0x7f492414ff40 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49240aae60 con 0x7f492414ff40 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49240b4310 con 0x7f4924149dc0 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49240afdd0 con 0x7f49240a9930 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.807+0000 7f492b855640 10 monclient(hunting): authenticate will time out at 1042.087043s 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.810+0000 7f49295f1640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f49240a9930 0x7f4924143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:57615/0 (socket says 172.21.15.55:57615) 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.810+0000 7f49295f1640 1 -- 172.21.15.55:0/2157771630 learned_addr learned my addr 172.21.15.55:0/2157771630 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317407627 0 0) 0x7f49240b4310 con 0x7f4924149dc0 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:48.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4910003520 con 0x7f4924149dc0 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843652316 0 0) 0x7f49240afdd0 con 0x7f49240a9930 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.811+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49240b4310 con 0x7f49240a9930 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1965425083 0 0) 0x7f4910003520 con 0x7f4924149dc0 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 >> v1:172.21.15.55:6790/0 conn(0x7f49240a9930 legacy=0x7f4924143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 >> v1:172.21.15.55:6789/0 conn(0x7f492414ff40 legacy=0x7f4924152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:48.822 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 1 monclient: found mon.noname-c 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492414a950 con 0x7f4924149dc0 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 10 monclient: _finish_auth 0 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4914001690 con 0x7f4924149dc0 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f491bfff640 10 monclient: dump: 2024-03-29T11:43:48.823 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f492b855640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.812+0000 7f492b855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:48.824 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.814+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49140017f0 con 0x7f4924149dc0 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.814+0000 7f491bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.814+0000 7f491bfff640 1 -- 172.21.15.55:0/2157771630 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4914001c40 con 0x7f4924149dc0 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.814+0000 7f491bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.814+0000 7f491bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.814+0000 7f491bfff640 10 monclient: dump: 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:48.825 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.814+0000 7f492b855640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.814+0000 7f492b855640 4 set_mon_vals no callback set 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.815+0000 7f492b855640 10 monclient: shutdown 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.815+0000 7f492b855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.815+0000 7f492b855640 1 -- 172.21.15.55:0/2157771630 >> v1:172.21.15.171:6789/0 conn(0x7f4924149dc0 legacy=0x7f492414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.815+0000 7f492b855640 1 -- 172.21.15.55:0/2157771630 shutdown_connections 2024-03-29T11:43:48.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.815+0000 7f492b855640 1 -- 172.21.15.55:0/2157771630 >> 172.21.15.55:0/2157771630 conn(0x7f49240b35e0 msgr2=0x7f49240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:48.828 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.816+0000 7f492b855640 1 -- 172.21.15.55:0/2157771630 shutdown_connections 2024-03-29T11:43:48.828 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.817+0000 7f492b855640 1 -- 172.21.15.55:0/2157771630 wait complete. 2024-03-29T11:43:48.828 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.818+0000 7f492b855640 10 monclient: build_initial_monmap 2024-03-29T11:43:48.828 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.818+0000 7f492b855640 10 monclient: monmap: 2024-03-29T11:43:48.828 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:48.828 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.820+0000 7f492b855640 1 librados: starting msgr at 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.820+0000 7f492b855640 1 librados: starting objecter 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.821+0000 7f492b855640 1 Processor -- start 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.821+0000 7f492b855640 1 -- start start 2024-03-29T11:43:48.829 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.821+0000 7f492b855640 1 librados: setting wanted keys 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.821+0000 7f492b855640 1 librados: calling monclient init 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.821+0000 7f492b855640 10 monclient: init 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.822+0000 7f492b855640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.822+0000 7f492b855640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.822+0000 7f492b855640 10 monclient(hunting): picked mon.noname-c con 0x7f49240a9930 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.823+0000 7f492b855640 10 monclient(hunting): picked mon.noname-b con 0x7f4924149dc0 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.823+0000 7f492b855640 10 monclient(hunting): picked mon.noname-a con 0x7f492414ff40 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.823+0000 7f492b855640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492414aae0 con 0x7f492414ff40 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.823+0000 7f492b855640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4924146b60 con 0x7f4924149dc0 2024-03-29T11:43:48.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.823+0000 7f492b855640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49241d6120 con 0x7f49240a9930 2024-03-29T11:43:48.831 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.823+0000 7f492b855640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:48.831 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.823+0000 7f492b855640 10 monclient(hunting): authenticate will time out at 1042.102863s 2024-03-29T11:43:48.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.824+0000 7f4928df0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4924149dc0 0x7f49241d53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:33292/0 (socket says 172.21.15.55:33292) 2024-03-29T11:43:48.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.825+0000 7f4928df0640 1 -- 172.21.15.55:0/281272079 learned_addr learned my addr 172.21.15.55:0/281272079 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.827+0000 7f4919ffb640 10 client.?.objecter ms_handle_connect 0x7f4924149dc0 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.827+0000 7f4919ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.827+0000 7f4919ffb640 10 client.?.objecter ms_handle_connect 0x7f49240a9930 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.827+0000 7f4919ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 client.?.objecter ms_handle_connect 0x7f492414ff40 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765294765 0 0) 0x7f49241d6120 con 0x7f49240a9930 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:43:48.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4908004eb0 con 0x7f49240a9930 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697596630 0 0) 0x7f492414aae0 con 0x7f492414ff40 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49241d6120 con 0x7f492414ff40 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1276754348 0 0) 0x7f4908004eb0 con 0x7f49240a9930 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f492414aae0 con 0x7f49240a9930 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f49200010a0 con 0x7f49240a9930 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:48.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): dump: 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:48.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3208300344 0 0) 0x7f49241d6120 con 0x7f492414ff40 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4908004eb0 con 0x7f492414ff40 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f491c001a70 con 0x7f492414ff40 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 10 monclient(hunting): dump: 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:48.837 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.828+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 968810559 0 0) 0x7f492414aae0 con 0x7f49240a9930 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 >> v1:172.21.15.171:6789/0 conn(0x7f4924149dc0 legacy=0x7f49241d53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 >> v1:172.21.15.55:6789/0 conn(0x7f492414ff40 legacy=0x7f49241d6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 1 monclient: found mon.c 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:48.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4924147750 con 0x7f49240a9930 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f4919ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f492b855640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f492b855640 10 monclient: _renew_subs 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f492b855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4924147a30 con 0x7f49240a9930 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f492b855640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.829+0000 7f492b855640 10 monclient: _renew_subs 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.830+0000 7f492b855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.830+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4924145890 con 0x7f49240a9930 2024-03-29T11:43:48.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.830+0000 7f492b855640 1 librados: init done 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.830+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4920001590 con 0x7f49240a9930 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.830+0000 7f4919ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.830+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4920015a20 con 0x7f49240a9930 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.830+0000 7f4919ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.830+0000 7f4919ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.830+0000 7f4919ffb640 10 monclient: dump: 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:48.840 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:48.841 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:48.841 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:48.841 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:48.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.831+0000 7f491b7fe640 4 set_mon_vals no callback set 2024-03-29T11:43:48.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.832+0000 7f492b855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:48.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.833+0000 7f492b855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.833+0000 7f492b855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.833+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4904005080 con 0x7f49240a9930 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.833+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f492002b9f0 con 0x7f49240a9930 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.834+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f4920095a40 con 0x7f49240a9930 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.834+0000 7f4919ffb640 10 client.4292.objecter ms_dispatch 0x7f4924000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.834+0000 7f4919ffb640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.834+0000 7f4919ffb640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.834+0000 7f4919ffb640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.834+0000 7f4928df0640 10 monclient: get_auth_request con 0x7f490806de90 auth_method 0 2024-03-29T11:43:48.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.835+0000 7f4919ffb640 10 client.4292.objecter ms_handle_connect 0x7f490806de90 2024-03-29T11:43:48.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.841+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4920061320 con 0x7f49240a9930 2024-03-29T11:43:48.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.841+0000 7f4919ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:48.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:48.841+0000 7f4919ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:49.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.171+0000 7f492b855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:49.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.171+0000 7f492b855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:49.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.171+0000 7f492b855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:43:49.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.171+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4904005330 con 0x7f49240a9930 2024-03-29T11:43:49.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.173+0000 7f4919ffb640 1 -- 172.21.15.55:0/281272079 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f4920063250 con 0x7f49240a9930 2024-03-29T11:43:49.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.173+0000 7f4919ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:49.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.173+0000 7f4919ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:49.177 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:43:49.177 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:49.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.176+0000 7f492b855640 10 librados: watch_flush enter 2024-03-29T11:43:49.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.177+0000 7f492b855640 10 librados: watch_flush exit 2024-03-29T11:43:49.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.177+0000 7f492b855640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:43:49.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.177+0000 7f492b855640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:43:49.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.177+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 >> v1:172.21.15.171:6816/1463197163 conn(0x7f490806de90 legacy=0x7f4908070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:49.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.177+0000 7f492b855640 10 monclient: shutdown 2024-03-29T11:43:49.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.177+0000 7f492b855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:49.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.177+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 >> v1:172.21.15.55:6790/0 conn(0x7f49240a9930 legacy=0x7f4924146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:49.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.177+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 shutdown_connections 2024-03-29T11:43:49.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.177+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 >> 172.21.15.55:0/281272079 conn(0x7f49240b35e0 msgr2=0x7f49240b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:49.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.178+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 shutdown_connections 2024-03-29T11:43:49.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.179+0000 7f492b855640 1 -- 172.21.15.55:0/281272079 wait complete. 2024-03-29T11:43:49.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:49.179+0000 7f492b855640 1 librados: shutdown 2024-03-29T11:43:49.199 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:56.008 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:56.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.162+0000 7f5df27f6640 10 monclient: get_monmap_and_config 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.162+0000 7f5df27f6640 10 monclient: build_initial_monmap 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.162+0000 7f5df27f6640 10 monclient: monmap: 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:43:56.162355+0000 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:43:56.162355+0000 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:43:56.166 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:56.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.163+0000 7f5df27f6640 1 Processor -- start 2024-03-29T11:43:56.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.163+0000 7f5df27f6640 1 -- start start 2024-03-29T11:43:56.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.163+0000 7f5df27f6640 10 monclient: init 2024-03-29T11:43:56.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 10 monclient(hunting): picked mon.noname-a con 0x7f5dec0ae250 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 10 monclient(hunting): picked mon.noname-c con 0x7f5dec13f970 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 10 monclient(hunting): picked mon.noname-b con 0x7f5dec142800 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0afe00 con 0x7f5dec0ae250 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec142c10 con 0x7f5dec13f970 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0afb20 con 0x7f5dec142800 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.164+0000 7f5df27f6640 10 monclient(hunting): authenticate will time out at 1049.442793s 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.167+0000 7f5df0d93640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f5dec142800 0x7f5dec14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:49911/0 (socket says 172.21.15.55:49911) 2024-03-29T11:43:56.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.167+0000 7f5df0d93640 1 -- 172.21.15.55:0/940596187 learned_addr learned my addr 172.21.15.55:0/940596187 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:56.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.167+0000 7f5deaffd640 1 -- 172.21.15.55:0/940596187 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394593349 0 0) 0x7f5dec0afb20 con 0x7f5dec142800 2024-03-29T11:43:56.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.167+0000 7f5deaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:56.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.167+0000 7f5deaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:56.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.167+0000 7f5deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:56.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.167+0000 7f5deaffd640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:43:56.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.167+0000 7f5deaffd640 1 -- 172.21.15.55:0/940596187 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd4003520 con 0x7f5dec142800 2024-03-29T11:43:56.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.168+0000 7f5deaffd640 1 -- 172.21.15.55:0/940596187 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 205065353 0 0) 0x7f5dd4003520 con 0x7f5dec142800 2024-03-29T11:43:56.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.168+0000 7f5deaffd640 1 -- 172.21.15.55:0/940596187 >> v1:172.21.15.171:6789/0 conn(0x7f5dec13f970 legacy=0x7f5dec1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:56.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.168+0000 7f5deaffd640 1 -- 172.21.15.55:0/940596187 >> v1:172.21.15.55:6789/0 conn(0x7f5dec0ae250 legacy=0x7f5dec13f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:56.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:56.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 1 monclient: found mon.noname-b 2024-03-29T11:43:56.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:56.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 1 -- 172.21.15.55:0/940596187 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec0aab80 con 0x7f5dec142800 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 1 -- 172.21.15.55:0/940596187 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5de0001690 con 0x7f5dec142800 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5deaffd640 10 monclient: dump: 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:56.174 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:56.175 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:56.175 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:56.175 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:56.175 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:56.175 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:56.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5df27f6640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.169+0000 7f5df27f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.170+0000 7f5deaffd640 1 -- 172.21.15.55:0/940596187 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5de0001920 con 0x7f5dec142800 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.170+0000 7f5deaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.170+0000 7f5df27f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.170+0000 7f5df27f6640 4 set_mon_vals no callback set 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.171+0000 7f5df27f6640 10 monclient: shutdown 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.171+0000 7f5df27f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.171+0000 7f5df27f6640 1 -- 172.21.15.55:0/940596187 >> v1:172.21.15.55:6790/0 conn(0x7f5dec142800 legacy=0x7f5dec14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.171+0000 7f5deaffd640 1 -- 172.21.15.55:0/940596187 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5de0001d30 con 0x7f5dec142800 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.171+0000 7f5deaffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.171+0000 7f5df27f6640 1 -- 172.21.15.55:0/940596187 shutdown_connections 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.171+0000 7f5df27f6640 1 -- 172.21.15.55:0/940596187 >> 172.21.15.55:0/940596187 conn(0x7f5dec0b35e0 msgr2=0x7f5dec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:56.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.171+0000 7f5df27f6640 1 -- 172.21.15.55:0/940596187 shutdown_connections 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.171+0000 7f5df27f6640 1 -- 172.21.15.55:0/940596187 wait complete. 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.172+0000 7f5df27f6640 10 monclient: build_initial_monmap 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.172+0000 7f5df27f6640 10 monclient: monmap: 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.173+0000 7f5df27f6640 1 librados: starting msgr at 2024-03-29T11:43:56.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.173+0000 7f5df27f6640 1 librados: starting objecter 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.173+0000 7f5df27f6640 1 Processor -- start 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.173+0000 7f5df27f6640 1 -- start start 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.173+0000 7f5df27f6640 1 librados: setting wanted keys 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.173+0000 7f5df27f6640 1 librados: calling monclient init 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.173+0000 7f5df27f6640 10 monclient: init 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.174+0000 7f5df27f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.174+0000 7f5df27f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.174+0000 7f5df27f6640 10 monclient(hunting): picked mon.noname-c con 0x7f5dec0ae250 addr v1:172.21.15.55:6790/0 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.175+0000 7f5df27f6640 10 monclient(hunting): picked mon.noname-b con 0x7f5dec13f970 addr v1:172.21.15.171:6789/0 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.175+0000 7f5df27f6640 10 monclient(hunting): picked mon.noname-a con 0x7f5dec142800 addr v1:172.21.15.55:6789/0 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.175+0000 7f5df27f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec144fa0 con 0x7f5dec142800 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.175+0000 7f5df27f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec138d00 con 0x7f5dec13f970 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.175+0000 7f5df27f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec145fa0 con 0x7f5dec0ae250 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.175+0000 7f5df27f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:56.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.175+0000 7f5df27f6640 10 monclient(hunting): authenticate will time out at 1049.454013s 2024-03-29T11:43:56.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.181+0000 7f5deb7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5dec13f970 0x7f5dec137ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:47499/0 (socket says 172.21.15.55:47499) 2024-03-29T11:43:56.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.181+0000 7f5deb7fe640 1 -- 172.21.15.55:0/237695386 learned_addr learned my addr 172.21.15.55:0/237695386 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:43:56.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 10 client.?.objecter ms_handle_connect 0x7f5dec13f970 2024-03-29T11:43:56.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:56.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 10 client.?.objecter ms_handle_connect 0x7f5dec142800 2024-03-29T11:43:56.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:56.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138408059 0 0) 0x7f5dec138d00 con 0x7f5dec13f970 2024-03-29T11:43:56.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:56.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:56.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:56.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:43:56.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.182+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd8004eb0 con 0x7f5dec13f970 2024-03-29T11:43:56.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1109604397 0 0) 0x7f5dd8004eb0 con 0x7f5dec13f970 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dec138d00 con 0x7f5dec13f970 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 client.?.objecter ms_handle_connect 0x7f5dec0ae250 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5de4001db0 con 0x7f5dec13f970 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:56.192 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926501409 0 0) 0x7f5dec144fa0 con 0x7f5dec142800 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:43:56.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd8004eb0 con 0x7f5dec142800 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610345537 0 0) 0x7f5dec145fa0 con 0x7f5dec0ae250 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.183+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dec144fa0 con 0x7f5dec0ae250 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 454178770 0 0) 0x7f5dd8004eb0 con 0x7f5dec142800 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dec145fa0 con 0x7f5dec142800 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5de00014b0 con 0x7f5dec142800 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:56.194 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 515516846 0 0) 0x7f5dec144fa0 con 0x7f5dec0ae250 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dd8004eb0 con 0x7f5dec0ae250 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 240387871 0 0) 0x7f5dec138d00 con 0x7f5dec13f970 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 >> v1:172.21.15.55:6790/0 conn(0x7f5dec0ae250 legacy=0x7f5dec13ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 >> v1:172.21.15.55:6789/0 conn(0x7f5dec142800 legacy=0x7f5dec1385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 monclient: found mon.b 2024-03-29T11:43:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec146280 con 0x7f5dec13f970 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5de8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5df27f6640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5df27f6640 10 monclient: _renew_subs 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5df27f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec145430 con 0x7f5dec13f970 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5df27f6640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5df27f6640 10 monclient: _renew_subs 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5df27f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec144350 con 0x7f5dec13f970 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.184+0000 7f5df27f6640 1 librados: init done 2024-03-29T11:43:56.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.185+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5de4001f10 con 0x7f5dec13f970 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.185+0000 7f5de8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.185+0000 7f5de9ffb640 4 set_mon_vals no callback set 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.186+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5de4003de0 con 0x7f5dec13f970 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.187+0000 7f5de8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.187+0000 7f5df27f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.188+0000 7f5de8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.188+0000 7f5de8ff9640 10 monclient: dump: 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:56.198 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.188+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f5de4004000 con 0x7f5dec13f970 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.188+0000 7f5df27f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.188+0000 7f5df27f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.188+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dcc005080 con 0x7f5dec13f970 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.189+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f5de4084ed0 con 0x7f5dec13f970 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.189+0000 7f5de8ff9640 10 client.4294.objecter ms_dispatch 0x7f5dec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.189+0000 7f5de8ff9640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.189+0000 7f5debfff640 10 monclient: get_auth_request con 0x7f5dd806e0d0 auth_method 0 2024-03-29T11:43:56.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.189+0000 7f5de8ff9640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:56.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.189+0000 7f5de8ff9640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T11:43:56.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.190+0000 7f5de8ff9640 10 client.4294.objecter ms_handle_connect 0x7f5dd806e0d0 2024-03-29T11:43:56.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.196+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5de40507b0 con 0x7f5dec13f970 2024-03-29T11:43:56.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.196+0000 7f5de8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:56.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.196+0000 7f5de8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:56.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.524+0000 7f5df27f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:56.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.524+0000 7f5df27f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:56.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.524+0000 7f5df27f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:43:56.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.524+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dcc005330 con 0x7f5dec13f970 2024-03-29T11:43:56.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.525+0000 7f5de8ff9640 1 -- 172.21.15.55:0/237695386 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f5de40526e0 con 0x7f5dec13f970 2024-03-29T11:43:56.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.525+0000 7f5de8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:56.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.525+0000 7f5de8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:56.531 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:43:56.532 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:56.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 10 librados: watch_flush enter 2024-03-29T11:43:56.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 10 librados: watch_flush exit 2024-03-29T11:43:56.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T11:43:56.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 10 client.4294.objecter successfully canceled tick 2024-03-29T11:43:56.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 >> v1:172.21.15.171:6816/1463197163 conn(0x7f5dd806e0d0 legacy=0x7f5dd8070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:56.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 10 monclient: shutdown 2024-03-29T11:43:56.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:56.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 >> v1:172.21.15.171:6789/0 conn(0x7f5dec13f970 legacy=0x7f5dec137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:56.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 shutdown_connections 2024-03-29T11:43:56.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.534+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 >> 172.21.15.55:0/237695386 conn(0x7f5dec0b35e0 msgr2=0x7f5dec0aa250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:56.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.537+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 shutdown_connections 2024-03-29T11:43:56.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.537+0000 7f5df27f6640 1 -- 172.21.15.55:0/237695386 wait complete. 2024-03-29T11:43:56.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:43:56.538+0000 7f5df27f6640 1 librados: shutdown 2024-03-29T11:43:56.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:03.367 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:03.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.520+0000 7fbca308d640 10 monclient: get_monmap_and_config 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.520+0000 7fbca308d640 10 monclient: build_initial_monmap 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.520+0000 7fbca308d640 10 monclient: monmap: 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:44:03.521238+0000 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:44:03.521238+0000 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:44:03.524 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.522+0000 7fbca308d640 1 Processor -- start 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 1 -- start start 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 10 monclient: init 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9c0b1f60 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 10 monclient(hunting): picked mon.noname-a con 0x7fbc9c071eb0 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9c143d10 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c06ef80 con 0x7fbc9c071eb0 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c06f260 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c0b40c0 con 0x7fbc9c143d10 2024-03-29T11:44:03.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:03.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.523+0000 7fbca308d640 10 monclient(hunting): authenticate will time out at 1056.803625s 2024-03-29T11:44:03.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.526+0000 7fbca0e29640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbc9c0b1f60 0x7fbc9c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:36639/0 (socket says 172.21.15.55:36639) 2024-03-29T11:44:03.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.526+0000 7fbca0e29640 1 -- 172.21.15.55:0/14283448 learned_addr learned my addr 172.21.15.55:0/14283448 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:03.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977053418 0 0) 0x7fbc9c06f260 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:03.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:03.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:03.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:44:03.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc84003520 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502608047 0 0) 0x7fbc9c0b40c0 con 0x7fbc9c143d10 2024-03-29T11:44:03.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:03.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:03.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:03.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:44:03.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.529+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc9c06f260 con 0x7fbc9c143d10 2024-03-29T11:44:03.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946428233 0 0) 0x7fbc9c06ef80 con 0x7fbc9c071eb0 2024-03-29T11:44:03.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:03.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:03.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc9c0b40c0 con 0x7fbc9c071eb0 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1507689742 0 0) 0x7fbc9c0b40c0 con 0x7fbc9c071eb0 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 >> v1:172.21.15.55:6790/0 conn(0x7fbc9c143d10 legacy=0x7fbc9c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 >> v1:172.21.15.171:6789/0 conn(0x7fbc9c0b1f60 legacy=0x7fbc9c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c155d70 con 0x7fbc9c071eb0 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fbc900015b0 con 0x7fbc9c071eb0 2024-03-29T11:44:03.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: dump: 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc90001840 con 0x7fbc9c071eb0 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:03.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 1 -- 172.21.15.55:0/14283448 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fbc90001bd0 con 0x7fbc9c071eb0 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.530+0000 7fbc9b7fe640 10 monclient: dump: 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.531+0000 7fbca308d640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.531+0000 7fbca308d640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:03.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.531+0000 7fbca308d640 4 set_mon_vals no callback set 2024-03-29T11:44:03.538 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.532+0000 7fbca308d640 10 monclient: shutdown 2024-03-29T11:44:03.538 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.532+0000 7fbca308d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:03.538 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.532+0000 7fbca308d640 1 -- 172.21.15.55:0/14283448 >> v1:172.21.15.55:6789/0 conn(0x7fbc9c071eb0 legacy=0x7fbc9c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:03.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.535+0000 7fbca308d640 1 -- 172.21.15.55:0/14283448 shutdown_connections 2024-03-29T11:44:03.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.535+0000 7fbca308d640 1 -- 172.21.15.55:0/14283448 >> 172.21.15.55:0/14283448 conn(0x7fbc9c066a00 msgr2=0x7fbc9c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:03.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.538+0000 7fbca308d640 1 -- 172.21.15.55:0/14283448 shutdown_connections 2024-03-29T11:44:03.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.539+0000 7fbca308d640 1 -- 172.21.15.55:0/14283448 wait complete. 2024-03-29T11:44:03.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.540+0000 7fbca308d640 10 monclient: build_initial_monmap 2024-03-29T11:44:03.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.540+0000 7fbca308d640 10 monclient: monmap: 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.540+0000 7fbca308d640 1 librados: starting msgr at 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.540+0000 7fbca308d640 1 librados: starting objecter 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.540+0000 7fbca308d640 1 Processor -- start 2024-03-29T11:44:03.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.540+0000 7fbca308d640 1 -- start start 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.540+0000 7fbca308d640 1 librados: setting wanted keys 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.540+0000 7fbca308d640 1 librados: calling monclient init 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.540+0000 7fbca308d640 10 monclient: init 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 10 monclient(hunting): picked mon.noname-a con 0x7fbc9c0b1f60 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9c143d10 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9c08b180 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c148850 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c14b3e0 con 0x7fbc9c08b180 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c14b650 con 0x7fbc9c143d10 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:03.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.541+0000 7fbca308d640 10 monclient(hunting): authenticate will time out at 1056.821188s 2024-03-29T11:44:03.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.548+0000 7fbca0e29640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fbc9c0b1f60 0x7fbc9c131210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:33035/0 (socket says 172.21.15.55:33035) 2024-03-29T11:44:03.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.548+0000 7fbca0e29640 1 -- 172.21.15.55:0/1754247963 learned_addr learned my addr 172.21.15.55:0/1754247963 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 client.?.objecter ms_handle_connect 0x7fbc9c0b1f60 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 client.?.objecter ms_handle_connect 0x7fbc9c08b180 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091478053 0 0) 0x7fbc9c148850 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc88004eb0 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186006598 0 0) 0x7fbc9c14b3e0 con 0x7fbc9c08b180 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc9c148850 con 0x7fbc9c08b180 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 client.?.objecter ms_handle_connect 0x7fbc9c143d10 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1474203111 0 0) 0x7fbc88004eb0 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.550+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc9c14b3e0 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fbc94001cd0 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient(hunting): dump: 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:03.558 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688034384 0 0) 0x7fbc9c14b650 con 0x7fbc9c143d10 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:44:03.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc88004eb0 con 0x7fbc9c143d10 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 885574784 0 0) 0x7fbc9c14b3e0 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 >> v1:172.21.15.55:6790/0 conn(0x7fbc9c143d10 legacy=0x7fbc9c148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 >> v1:172.21.15.171:6789/0 conn(0x7fbc9c08b180 legacy=0x7fbc9c12b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 1 monclient: found mon.a 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c130cf0 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient: _finish_auth 0 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.551+0000 7fbc997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.552+0000 7fbca308d640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.552+0000 7fbca308d640 10 monclient: _renew_subs 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.552+0000 7fbca308d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:03.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.552+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c130980 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.552+0000 7fbca308d640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:03.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.552+0000 7fbca308d640 10 monclient: _renew_subs 2024-03-29T11:44:03.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.552+0000 7fbca308d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:03.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.552+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c154da0 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.552+0000 7fbca308d640 1 librados: init done 2024-03-29T11:44:03.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.554+0000 7fbca308d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:03.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.554+0000 7fbca308d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:03.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.555+0000 7fbca308d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:03.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.555+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c130f30 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.556+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc94004b60 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.556+0000 7fbc997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:03.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.556+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fbc94004e70 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.556+0000 7fbc997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.556+0000 7fbc997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.556+0000 7fbc997fa640 10 monclient: dump: 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.556+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fbc9401ae40 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.557+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fbc94084f40 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.557+0000 7fbc997fa640 10 client.4299.objecter ms_dispatch 0x7fbc9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:03.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.557+0000 7fbc997fa640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:03.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.557+0000 7fbc997fa640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:03.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.557+0000 7fbc997fa640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T11:44:03.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.557+0000 7fbc9bfff640 10 monclient: get_auth_request con 0x7fbc8806df10 auth_method 0 2024-03-29T11:44:03.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.558+0000 7fbc9affd640 4 set_mon_vals no callback set 2024-03-29T11:44:03.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.558+0000 7fbc997fa640 10 client.4299.objecter ms_handle_connect 0x7fbc8806df10 2024-03-29T11:44:03.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.565+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc9c130f30 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.565+0000 7fbc997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:03.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.565+0000 7fbc997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:03.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.889+0000 7fbca308d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:03.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.889+0000 7fbca308d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:03.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.889+0000 7fbca308d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:03.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.889+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc9c130f30 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.890+0000 7fbc997fa640 1 -- 172.21.15.55:0/1754247963 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fbc9c130f30 con 0x7fbc9c0b1f60 2024-03-29T11:44:03.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.890+0000 7fbc997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:03.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.890+0000 7fbc997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:03.895 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:44:03.895 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:03.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.894+0000 7fbca308d640 10 librados: watch_flush enter 2024-03-29T11:44:03.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.894+0000 7fbca308d640 10 librados: watch_flush exit 2024-03-29T11:44:03.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.894+0000 7fbca308d640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T11:44:03.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.894+0000 7fbca308d640 10 client.4299.objecter successfully canceled tick 2024-03-29T11:44:03.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.895+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 >> v1:172.21.15.171:6816/1463197163 conn(0x7fbc8806df10 legacy=0x7fbc880703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:03.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.895+0000 7fbca308d640 10 monclient: shutdown 2024-03-29T11:44:03.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.895+0000 7fbca308d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:03.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.895+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 >> v1:172.21.15.55:6789/0 conn(0x7fbc9c0b1f60 legacy=0x7fbc9c131210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:03.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.896+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 shutdown_connections 2024-03-29T11:44:03.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.896+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 >> 172.21.15.55:0/1754247963 conn(0x7fbc9c066a00 msgr2=0x7fbc9c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:03.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.899+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 shutdown_connections 2024-03-29T11:44:03.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.899+0000 7fbca308d640 1 -- 172.21.15.55:0/1754247963 wait complete. 2024-03-29T11:44:03.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:03.899+0000 7fbca308d640 1 librados: shutdown 2024-03-29T11:44:03.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:10.729 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:10.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.879+0000 7fd33ad41640 10 monclient: get_monmap_and_config 2024-03-29T11:44:10.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.879+0000 7fd33ad41640 10 monclient: build_initial_monmap 2024-03-29T11:44:10.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.880+0000 7fd33ad41640 10 monclient: monmap: 2024-03-29T11:44:10.884 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:10.884 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:10.884 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:44:10.881153+0000 2024-03-29T11:44:10.885 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:44:10.881153+0000 2024-03-29T11:44:10.885 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:10.885 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:10.885 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:10.885 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:44:10.885 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:44:10.885 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:10.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.881+0000 7fd33ad41640 1 Processor -- start 2024-03-29T11:44:10.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.881+0000 7fd33ad41640 1 -- start start 2024-03-29T11:44:10.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.881+0000 7fd33ad41640 10 monclient: init 2024-03-29T11:44:10.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.882+0000 7fd33ad41640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:10.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.883+0000 7fd33ad41640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:10.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.883+0000 7fd33ad41640 10 monclient(hunting): picked mon.noname-c con 0x7fd32c006890 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:10.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.888+0000 7fd33ad41640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c006c70 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:10.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.888+0000 7fd339d3f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd32c006890 0x7fd32c0c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:55521/0 (socket says 172.21.15.55:55521) 2024-03-29T11:44:10.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.888+0000 7fd339d3f640 1 -- 172.21.15.55:0/608144232 learned_addr learned my addr 172.21.15.55:0/608144232 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:10.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.888+0000 7fd33ad41640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c02b4c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:10.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.888+0000 7fd33ad41640 1 -- 172.21.15.55:0/608144232 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c02c830 con 0x7fd32c02b4c0 2024-03-29T11:44:10.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.889+0000 7fd33ad41640 1 -- 172.21.15.55:0/608144232 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c02afa0 con 0x7fd32c006890 2024-03-29T11:44:10.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.889+0000 7fd33ad41640 1 -- 172.21.15.55:0/608144232 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c02aba0 con 0x7fd32c006c70 2024-03-29T11:44:10.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.889+0000 7fd33ad41640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:10.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.889+0000 7fd33ad41640 10 monclient(hunting): authenticate will time out at 1064.169182s 2024-03-29T11:44:10.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789605295 0 0) 0x7fd32c02afa0 con 0x7fd32c006890 2024-03-29T11:44:10.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 10 monclient(hunting): _negotiate 2024-03-29T11:44:10.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:10.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:10.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:44:10.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd320003520 con 0x7fd32c006890 2024-03-29T11:44:10.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850261073 0 0) 0x7fd32c02aba0 con 0x7fd32c006c70 2024-03-29T11:44:10.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 10 monclient(hunting): _negotiate 2024-03-29T11:44:10.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:10.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:10.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.891+0000 7fd338d3d640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:44:10.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.892+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c02afa0 con 0x7fd32c006c70 2024-03-29T11:44:10.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3385228003 0 0) 0x7fd320003520 con 0x7fd32c006890 2024-03-29T11:44:10.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 >> v1:172.21.15.55:6790/0 conn(0x7fd32c006c70 legacy=0x7fd32c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 >> v1:172.21.15.55:6789/0 conn(0x7fd32c02b4c0 legacy=0x7fd32c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 10 monclient: _finish_hunting 0 2024-03-29T11:44:10.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 1 monclient: found mon.noname-c 2024-03-29T11:44:10.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:10.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c02ad30 con 0x7fd32c006890 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 10 monclient: _finish_auth 0 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 10 monclient: _check_auth_tickets 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd33404f730 con 0x7fd32c006890 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.894+0000 7fd338d3d640 10 monclient: dump: 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:10.900 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.895+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3340008d0 con 0x7fd32c006890 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.895+0000 7fd338d3d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.895+0000 7fd338d3d640 1 -- 172.21.15.55:0/608144232 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd3340758b0 con 0x7fd32c006890 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.895+0000 7fd338d3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.895+0000 7fd338d3d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.895+0000 7fd338d3d640 10 monclient: dump: 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:10.901 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:10.902 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:10.902 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:10.902 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:10.902 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:10.902 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:10.902 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:10.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.895+0000 7fd33ad41640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.895+0000 7fd33ad41640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.895+0000 7fd33ad41640 4 set_mon_vals no callback set 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.896+0000 7fd33ad41640 10 monclient: shutdown 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.896+0000 7fd33ad41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.896+0000 7fd33ad41640 1 -- 172.21.15.55:0/608144232 >> v1:172.21.15.171:6789/0 conn(0x7fd32c006890 legacy=0x7fd32c0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.896+0000 7fd33ad41640 1 -- 172.21.15.55:0/608144232 shutdown_connections 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.896+0000 7fd33ad41640 1 -- 172.21.15.55:0/608144232 >> 172.21.15.55:0/608144232 conn(0x7fd32c0058f0 msgr2=0x7fd32c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.897+0000 7fd33ad41640 1 -- 172.21.15.55:0/608144232 shutdown_connections 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.897+0000 7fd33ad41640 1 -- 172.21.15.55:0/608144232 wait complete. 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.898+0000 7fd33ad41640 10 monclient: build_initial_monmap 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.898+0000 7fd33ad41640 10 monclient: monmap: 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:10.904 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.898+0000 7fd33ad41640 1 librados: starting msgr at 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.898+0000 7fd33ad41640 1 librados: starting objecter 2024-03-29T11:44:10.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.899+0000 7fd33ad41640 1 Processor -- start 2024-03-29T11:44:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.900+0000 7fd33ad41640 1 -- start start 2024-03-29T11:44:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.900+0000 7fd33ad41640 1 librados: setting wanted keys 2024-03-29T11:44:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.900+0000 7fd33ad41640 1 librados: calling monclient init 2024-03-29T11:44:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.900+0000 7fd33ad41640 10 monclient: init 2024-03-29T11:44:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c006890 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:10.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 10 monclient(hunting): picked mon.noname-c con 0x7fd32c02b4c0 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c02d820 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c02ece0 con 0x7fd32c02d820 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c02f930 con 0x7fd32c006890 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c02fc10 con 0x7fd32c02b4c0 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.901+0000 7fd33ad41640 10 monclient(hunting): authenticate will time out at 1064.181124s 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd33953e640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fd32c02b4c0 0x7fd32c0305c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:38463/0 (socket says 172.21.15.55:38463) 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd33953e640 1 -- 172.21.15.55:0/4110711440 learned_addr learned my addr 172.21.15.55:0/4110711440 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd32a7fc640 10 client.?.objecter ms_handle_connect 0x7fd32c02b4c0 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd32a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474681765 0 0) 0x7fd32c02fc10 con 0x7fd32c02b4c0 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd32a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:10.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd32a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:10.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd32a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:10.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd32a7fc640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:44:10.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.905+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd318004eb0 con 0x7fd32c02b4c0 2024-03-29T11:44:10.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.908+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3733298946 0 0) 0x7fd318004eb0 con 0x7fd32c02b4c0 2024-03-29T11:44:10.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.908+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd32c02fc10 con 0x7fd32c02b4c0 2024-03-29T11:44:10.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.908+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd324001a70 con 0x7fd32c02b4c0 2024-03-29T11:44:10.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.908+0000 7fd32a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:10.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.908+0000 7fd32a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:10.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.908+0000 7fd32a7fc640 10 monclient(hunting): dump: 2024-03-29T11:44:10.918 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:10.918 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:10.918 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:10.918 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:10.918 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:10.918 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:10.918 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:10.919 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:10.919 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:10.919 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:10.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.908+0000 7fd32a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:10.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.908+0000 7fd32a7fc640 10 client.?.objecter ms_handle_connect 0x7fd32c006890 2024-03-29T11:44:10.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.908+0000 7fd32a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:10.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 client.?.objecter ms_handle_connect 0x7fd32c02d820 2024-03-29T11:44:10.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:10.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2637652484 0 0) 0x7fd32c02fc10 con 0x7fd32c02b4c0 2024-03-29T11:44:10.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 >> v1:172.21.15.171:6789/0 conn(0x7fd32c006890 legacy=0x7fd32c02e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 >> v1:172.21.15.55:6789/0 conn(0x7fd32c02d820 legacy=0x7fd32c030cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:10.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 1 monclient: found mon.c 2024-03-29T11:44:10.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:10.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:10.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c16faf0 con 0x7fd32c02b4c0 2024-03-29T11:44:10.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:10.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:10.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:10.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd324001cd0 con 0x7fd32c02b4c0 2024-03-29T11:44:10.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:10.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd324003a40 con 0x7fd32c02b4c0 2024-03-29T11:44:10.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:10.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:10.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32a7fc640 10 monclient: dump: 2024-03-29T11:44:10.922 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:10.922 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:10.922 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:10.922 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:10.922 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:10.922 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:10.923 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:10.923 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:10.923 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:10.923 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:10.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.909+0000 7fd32bfff640 4 set_mon_vals no callback set 2024-03-29T11:44:10.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.910+0000 7fd33ad41640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:44:10.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.910+0000 7fd33ad41640 10 monclient: _renew_subs 2024-03-29T11:44:10.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.910+0000 7fd33ad41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:10.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.910+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c16cac0 con 0x7fd32c02b4c0 2024-03-29T11:44:10.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.910+0000 7fd33ad41640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:10.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.910+0000 7fd33ad41640 10 monclient: _renew_subs 2024-03-29T11:44:10.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.910+0000 7fd33ad41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:10.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.910+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c16cfa0 con 0x7fd32c02b4c0 2024-03-29T11:44:10.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.910+0000 7fd33ad41640 1 librados: init done 2024-03-29T11:44:10.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.911+0000 7fd33ad41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:10.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.912+0000 7fd33ad41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:10.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.912+0000 7fd33ad41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:10.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.912+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c0c49b0 con 0x7fd32c02b4c0 2024-03-29T11:44:10.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.913+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fd324003ba0 con 0x7fd32c02b4c0 2024-03-29T11:44:10.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.913+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fd324084b20 con 0x7fd32c02b4c0 2024-03-29T11:44:10.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.913+0000 7fd32a7fc640 10 client.4310.objecter ms_dispatch 0x7fd334000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:10.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.913+0000 7fd32a7fc640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:10.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.913+0000 7fd32a7fc640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:10.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.913+0000 7fd32a7fc640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T11:44:10.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.914+0000 7fd339d3f640 10 monclient: get_auth_request con 0x7fd31806dc30 auth_method 0 2024-03-29T11:44:10.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.914+0000 7fd32a7fc640 10 client.4310.objecter ms_handle_connect 0x7fd31806dc30 2024-03-29T11:44:10.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.922+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd324050230 con 0x7fd32c02b4c0 2024-03-29T11:44:10.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.923+0000 7fd32a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:10.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:10.923+0000 7fd32a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:11.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.245+0000 7fd33ad41640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:11.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.245+0000 7fd33ad41640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:11.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.245+0000 7fd33ad41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:11.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.245+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd32c02a580 con 0x7fd32c02b4c0 2024-03-29T11:44:11.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.247+0000 7fd32a7fc640 1 -- 172.21.15.55:0/4110711440 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fd32c02a580 con 0x7fd32c02b4c0 2024-03-29T11:44:11.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.247+0000 7fd32a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:11.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.247+0000 7fd32a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:11.253 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:44:11.253 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:11.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.251+0000 7fd33ad41640 10 librados: watch_flush enter 2024-03-29T11:44:11.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.251+0000 7fd33ad41640 10 librados: watch_flush exit 2024-03-29T11:44:11.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.251+0000 7fd33ad41640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T11:44:11.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.251+0000 7fd33ad41640 10 client.4310.objecter successfully canceled tick 2024-03-29T11:44:11.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.251+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 >> v1:172.21.15.171:6816/1463197163 conn(0x7fd31806dc30 legacy=0x7fd3180700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:11.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.251+0000 7fd33ad41640 10 monclient: shutdown 2024-03-29T11:44:11.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.251+0000 7fd33ad41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:11.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.251+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 >> v1:172.21.15.55:6790/0 conn(0x7fd32c02b4c0 legacy=0x7fd32c0305c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:11.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.254+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 shutdown_connections 2024-03-29T11:44:11.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.254+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 >> 172.21.15.55:0/4110711440 conn(0x7fd32c0058f0 msgr2=0x7fd32c038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:11.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.254+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 shutdown_connections 2024-03-29T11:44:11.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.255+0000 7fd33ad41640 1 -- 172.21.15.55:0/4110711440 wait complete. 2024-03-29T11:44:11.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:11.255+0000 7fd33ad41640 1 librados: shutdown 2024-03-29T11:44:11.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:18.085 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:18.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.230+0000 7f65e59de640 10 monclient: get_monmap_and_config 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.230+0000 7f65e59de640 10 monclient: build_initial_monmap 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.230+0000 7f65e59de640 10 monclient: monmap: 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:44:18.231644+0000 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:44:18.231644+0000 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:44:18.236 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:44:18.237 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:18.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.231+0000 7f65e59de640 1 Processor -- start 2024-03-29T11:44:18.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.232+0000 7f65e59de640 1 -- start start 2024-03-29T11:44:18.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.232+0000 7f65e59de640 10 monclient: init 2024-03-29T11:44:18.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.232+0000 7f65e59de640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:18.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.232+0000 7f65e59de640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:18.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.232+0000 7f65e59de640 10 monclient(hunting): picked mon.noname-a con 0x7f65e00acfa0 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:18.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.233+0000 7f65e59de640 10 monclient(hunting): picked mon.noname-b con 0x7f65e0072f10 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:18.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.233+0000 7f65e59de640 10 monclient(hunting): picked mon.noname-c con 0x7f65e00ad380 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:18.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.233+0000 7f65e59de640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00aa9f0 con 0x7f65e00acfa0 2024-03-29T11:44:18.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.233+0000 7f65e59de640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00ac4c0 con 0x7f65e00ad380 2024-03-29T11:44:18.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.233+0000 7f65e59de640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00ac7a0 con 0x7f65e0072f10 2024-03-29T11:44:18.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.233+0000 7f65e59de640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:18.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.233+0000 7f65e59de640 10 monclient(hunting): authenticate will time out at 1071.512700s 2024-03-29T11:44:18.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.237+0000 7f65dffff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f65e0072f10 0x7f65e0141d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:42081/0 (socket says 172.21.15.55:42081) 2024-03-29T11:44:18.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.237+0000 7f65dffff640 1 -- 172.21.15.55:0/538636271 learned_addr learned my addr 172.21.15.55:0/538636271 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:18.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 1 -- 172.21.15.55:0/538636271 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256346398 0 0) 0x7f65e00ac7a0 con 0x7f65e0072f10 2024-03-29T11:44:18.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:18.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:18.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:18.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:44:18.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 1 -- 172.21.15.55:0/538636271 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c8003520 con 0x7f65e0072f10 2024-03-29T11:44:18.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 1 -- 172.21.15.55:0/538636271 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1017174106 0 0) 0x7f65c8003520 con 0x7f65e0072f10 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 1 -- 172.21.15.55:0/538636271 >> v1:172.21.15.171:6789/0 conn(0x7f65e00ad380 legacy=0x7f65e01497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 1 -- 172.21.15.55:0/538636271 >> v1:172.21.15.55:6789/0 conn(0x7f65e00acfa0 legacy=0x7f65e0141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 1 monclient: found mon.noname-b 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 1 -- 172.21.15.55:0/538636271 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e00ab5a0 con 0x7f65e0072f10 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 1 -- 172.21.15.55:0/538636271 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f65d4004a10 con 0x7f65e0072f10 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:18.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.238+0000 7f65df7fe640 10 monclient: dump: 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.239+0000 7f65e59de640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:44:18.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.239+0000 7f65e59de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:18.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.240+0000 7f65df7fe640 1 -- 172.21.15.55:0/538636271 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65d4004c70 con 0x7f65e0072f10 2024-03-29T11:44:18.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.240+0000 7f65df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:18.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.240+0000 7f65df7fe640 1 -- 172.21.15.55:0/538636271 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f65d4004a10 con 0x7f65e0072f10 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.240+0000 7f65df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.240+0000 7f65df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.240+0000 7f65df7fe640 10 monclient: dump: 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.240+0000 7f65e59de640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:18.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.240+0000 7f65e59de640 4 set_mon_vals no callback set 2024-03-29T11:44:18.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.241+0000 7f65e59de640 10 monclient: shutdown 2024-03-29T11:44:18.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.241+0000 7f65e59de640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:18.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.241+0000 7f65e59de640 1 -- 172.21.15.55:0/538636271 >> v1:172.21.15.55:6790/0 conn(0x7f65e0072f10 legacy=0x7f65e0141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:18.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.242+0000 7f65e59de640 1 -- 172.21.15.55:0/538636271 shutdown_connections 2024-03-29T11:44:18.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.242+0000 7f65e59de640 1 -- 172.21.15.55:0/538636271 >> 172.21.15.55:0/538636271 conn(0x7f65e00b3a30 msgr2=0x7f65e00b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:18.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.242+0000 7f65e59de640 1 -- 172.21.15.55:0/538636271 shutdown_connections 2024-03-29T11:44:18.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.242+0000 7f65e59de640 1 -- 172.21.15.55:0/538636271 wait complete. 2024-03-29T11:44:18.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.243+0000 7f65e59de640 10 monclient: build_initial_monmap 2024-03-29T11:44:18.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.243+0000 7f65e59de640 10 monclient: monmap: 2024-03-29T11:44:18.249 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:18.249 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:18.249 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:44:18.249 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:44:18.250 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:18.250 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:18.250 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:18.250 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:44:18.250 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:44:18.250 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:18.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.244+0000 7f65e59de640 1 librados: starting msgr at 2024-03-29T11:44:18.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.244+0000 7f65e59de640 1 librados: starting objecter 2024-03-29T11:44:18.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.244+0000 7f65e59de640 1 Processor -- start 2024-03-29T11:44:18.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.244+0000 7f65e59de640 1 -- start start 2024-03-29T11:44:18.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.244+0000 7f65e59de640 1 librados: setting wanted keys 2024-03-29T11:44:18.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.244+0000 7f65e59de640 1 librados: calling monclient init 2024-03-29T11:44:18.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.244+0000 7f65e59de640 10 monclient: init 2024-03-29T11:44:18.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.245+0000 7f65e59de640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:18.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.245+0000 7f65e59de640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:18.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.245+0000 7f65e59de640 10 monclient(hunting): picked mon.noname-b con 0x7f65e0072f10 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:18.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.245+0000 7f65e59de640 10 monclient(hunting): picked mon.noname-a con 0x7f65e00acfa0 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:18.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.245+0000 7f65e59de640 10 monclient(hunting): picked mon.noname-c con 0x7f65e00ad380 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:18.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65e49dc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f65e0072f10 0x7f65e013cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:59659/0 (socket says 172.21.15.55:59659) 2024-03-29T11:44:18.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65e49dc640 1 -- 172.21.15.55:0/310716054 learned_addr learned my addr 172.21.15.55:0/310716054 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:18.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.245+0000 7f65e59de640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e013ee30 con 0x7f65e00acfa0 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e0139500 con 0x7f65e0072f10 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e013e020 con 0x7f65e00ad380 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65e59de640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65e59de640 10 monclient(hunting): authenticate will time out at 1071.528996s 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65dd7fa640 10 client.?.objecter ms_handle_connect 0x7f65e00ad380 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65dd7fa640 10 client.?.objecter ms_handle_connect 0x7f65e0072f10 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.249+0000 7f65dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.250+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838430067 0 0) 0x7f65e013e020 con 0x7f65e00ad380 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.250+0000 7f65dd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.250+0000 7f65dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.250+0000 7f65dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.250+0000 7f65dd7fa640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:44:18.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.250+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65cc004eb0 con 0x7f65e00ad380 2024-03-29T11:44:18.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167938997 0 0) 0x7f65e0139500 con 0x7f65e0072f10 2024-03-29T11:44:18.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:18.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65cc0051d0 con 0x7f65e0072f10 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 626925159 0 0) 0x7f65cc004eb0 con 0x7f65e00ad380 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65cc005640 con 0x7f65e00ad380 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f65d8001a90 con 0x7f65e00ad380 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 monclient(hunting): dump: 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:18.258 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 client.?.objecter ms_handle_connect 0x7f65e00acfa0 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.251+0000 7f65dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.252+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537768553 0 0) 0x7f65e013ee30 con 0x7f65e00acfa0 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.252+0000 7f65dd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:18.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.252+0000 7f65dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:18.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.252+0000 7f65dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:18.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.252+0000 7f65dd7fa640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:44:18.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.252+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65cc005c10 con 0x7f65e00acfa0 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.254+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 449024549 0 0) 0x7f65cc005c10 con 0x7f65e00acfa0 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.254+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65e013ee30 con 0x7f65e00acfa0 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.254+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f65d4004a10 con 0x7f65e00acfa0 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.254+0000 7f65dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.254+0000 7f65dd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.254+0000 7f65dd7fa640 10 monclient(hunting): dump: 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.264 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.254+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2497108554 0 0) 0x7f65cc0051d0 con 0x7f65e0072f10 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.254+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65cc008580 con 0x7f65e0072f10 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.254+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f65d00018e0 con 0x7f65e0072f10 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 10 monclient(hunting): dump: 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:18.265 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 370526467 0 0) 0x7f65cc005640 con 0x7f65e00ad380 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 >> v1:172.21.15.171:6789/0 conn(0x7f65e0072f10 legacy=0x7f65e013cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 >> v1:172.21.15.55:6789/0 conn(0x7f65e00acfa0 legacy=0x7f65e01db640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:18.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 1 monclient: found mon.c 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e013c8a0 con 0x7f65e00ad380 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65e59de640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65e59de640 10 monclient: _renew_subs 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65e59de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e013bfc0 con 0x7f65e00ad380 2024-03-29T11:44:18.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65e59de640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65e59de640 10 monclient: _renew_subs 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65e59de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e0071950 con 0x7f65e00ad380 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65e59de640 1 librados: init done 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65d8001fa0 con 0x7f65e00ad380 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65deffd640 4 set_mon_vals no callback set 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f65d8001a90 con 0x7f65e00ad380 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.255+0000 7f65dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.256+0000 7f65dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.256+0000 7f65dd7fa640 10 monclient: dump: 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:18.268 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.257+0000 7f65e59de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.257+0000 7f65e59de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.257+0000 7f65e59de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.257+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c0005080 con 0x7f65e00ad380 2024-03-29T11:44:18.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.259+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f65d80197e0 con 0x7f65e00ad380 2024-03-29T11:44:18.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.259+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f65d8083460 con 0x7f65e00ad380 2024-03-29T11:44:18.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.259+0000 7f65dd7fa640 10 client.4316.objecter ms_dispatch 0x7f65e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:18.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.259+0000 7f65dd7fa640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:18.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.259+0000 7f65dd7fa640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:18.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.259+0000 7f65dd7fa640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:44:18.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.259+0000 7f65e49dc640 10 monclient: get_auth_request con 0x7f65cc06ede0 auth_method 0 2024-03-29T11:44:18.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.260+0000 7f65dd7fa640 10 client.4316.objecter ms_handle_connect 0x7f65cc06ede0 2024-03-29T11:44:18.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.269+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65d804ecb0 con 0x7f65e00ad380 2024-03-29T11:44:18.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.269+0000 7f65dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:18.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.269+0000 7f65dd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:18.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.599+0000 7f65e59de640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:18.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.600+0000 7f65e59de640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:18.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.600+0000 7f65e59de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:18.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.600+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65c0005330 con 0x7f65e00ad380 2024-03-29T11:44:18.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.601+0000 7f65dd7fa640 1 -- 172.21.15.55:0/310716054 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f65d8050db0 con 0x7f65e00ad380 2024-03-29T11:44:18.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.601+0000 7f65dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:18.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.601+0000 7f65dd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:18.606 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:44:18.606 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:18.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.605+0000 7f65e59de640 10 librados: watch_flush enter 2024-03-29T11:44:18.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.605+0000 7f65e59de640 10 librados: watch_flush exit 2024-03-29T11:44:18.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.605+0000 7f65e59de640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:44:18.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.605+0000 7f65e59de640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:44:18.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.607+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 >> v1:172.21.15.171:6816/1463197163 conn(0x7f65cc06ede0 legacy=0x7f65cc0712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:18.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.607+0000 7f65e59de640 10 monclient: shutdown 2024-03-29T11:44:18.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.607+0000 7f65e59de640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:18.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.607+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 >> v1:172.21.15.55:6790/0 conn(0x7f65e00ad380 legacy=0x7f65e013e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:18.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.610+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 shutdown_connections 2024-03-29T11:44:18.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.610+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 >> 172.21.15.55:0/310716054 conn(0x7f65e00b3a30 msgr2=0x7f65e00b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:18.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.613+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 shutdown_connections 2024-03-29T11:44:18.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.614+0000 7f65e59de640 1 -- 172.21.15.55:0/310716054 wait complete. 2024-03-29T11:44:18.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:18.614+0000 7f65e59de640 1 librados: shutdown 2024-03-29T11:44:18.637 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:25.444 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:25.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.593+0000 7fc3c67c7640 10 monclient: get_monmap_and_config 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.593+0000 7fc3c67c7640 10 monclient: build_initial_monmap 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.593+0000 7fc3c67c7640 10 monclient: monmap: 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:44:25.594988+0000 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:44:25.594988+0000 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:44:25.598 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:25.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.595+0000 7fc3c67c7640 1 Processor -- start 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.595+0000 7fc3c67c7640 1 -- start start 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.595+0000 7fc3c67c7640 10 monclient: init 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.596+0000 7fc3c67c7640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.598+0000 7fc3c67c7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.598+0000 7fc3c67c7640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c00b1f60 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.598+0000 7fc3c67c7640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c0071eb0 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.598+0000 7fc3c67c7640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c0143d10 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.598+0000 7fc3c67c7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c006ef80 con 0x7fc3c00b1f60 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.598+0000 7fc3c67c7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c006f260 con 0x7fc3c0143d10 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.598+0000 7fc3c67c7640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00b40c0 con 0x7fc3c0071eb0 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.598+0000 7fc3c67c7640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:25.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.598+0000 7fc3c67c7640 10 monclient(hunting): authenticate will time out at 1078.877977s 2024-03-29T11:44:25.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.602+0000 7fc3bf7fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fc3c0071eb0 0x7fc3c0146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:58917/0 (socket says 172.21.15.55:58917) 2024-03-29T11:44:25.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.602+0000 7fc3bf7fe640 1 -- 172.21.15.55:0/492657359 learned_addr learned my addr 172.21.15.55:0/492657359 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:25.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 1 -- 172.21.15.55:0/492657359 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910676993 0 0) 0x7fc3c00b40c0 con 0x7fc3c0071eb0 2024-03-29T11:44:25.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:25.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:25.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:25.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:44:25.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 1 -- 172.21.15.55:0/492657359 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a8003520 con 0x7fc3c0071eb0 2024-03-29T11:44:25.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 1 -- 172.21.15.55:0/492657359 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1164632384 0 0) 0x7fc3a8003520 con 0x7fc3c0071eb0 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 1 -- 172.21.15.55:0/492657359 >> v1:172.21.15.171:6789/0 conn(0x7fc3c0143d10 legacy=0x7fc3c0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 1 -- 172.21.15.55:0/492657359 >> v1:172.21.15.55:6789/0 conn(0x7fc3c00b1f60 legacy=0x7fc3c0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 1 monclient: found mon.noname-b 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 1 -- 172.21.15.55:0/492657359 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c0155d70 con 0x7fc3c0071eb0 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 1 -- 172.21.15.55:0/492657359 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc3b4001670 con 0x7fc3c0071eb0 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:25.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3beffd640 10 monclient: dump: 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3c67c7640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:44:25.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.603+0000 7fc3c67c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:25.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.604+0000 7fc3beffd640 1 -- 172.21.15.55:0/492657359 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3b40018d0 con 0x7fc3c0071eb0 2024-03-29T11:44:25.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.604+0000 7fc3beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:25.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.604+0000 7fc3c67c7640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:25.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.604+0000 7fc3c67c7640 4 set_mon_vals no callback set 2024-03-29T11:44:25.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.605+0000 7fc3c67c7640 10 monclient: shutdown 2024-03-29T11:44:25.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.605+0000 7fc3c67c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:25.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.605+0000 7fc3c67c7640 1 -- 172.21.15.55:0/492657359 >> v1:172.21.15.55:6790/0 conn(0x7fc3c0071eb0 legacy=0x7fc3c0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:25.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.605+0000 7fc3beffd640 1 -- 172.21.15.55:0/492657359 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc3b4001670 con 0x7fc3c0071eb0 2024-03-29T11:44:25.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.605+0000 7fc3beffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:44:25.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.605+0000 7fc3c67c7640 1 -- 172.21.15.55:0/492657359 shutdown_connections 2024-03-29T11:44:25.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.605+0000 7fc3c67c7640 1 -- 172.21.15.55:0/492657359 >> 172.21.15.55:0/492657359 conn(0x7fc3c0066a00 msgr2=0x7fc3c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:25.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.606+0000 7fc3c67c7640 1 -- 172.21.15.55:0/492657359 shutdown_connections 2024-03-29T11:44:25.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.606+0000 7fc3c67c7640 1 -- 172.21.15.55:0/492657359 wait complete. 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.607+0000 7fc3c67c7640 10 monclient: build_initial_monmap 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.607+0000 7fc3c67c7640 10 monclient: monmap: 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.608+0000 7fc3c67c7640 1 librados: starting msgr at 2024-03-29T11:44:25.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.608+0000 7fc3c67c7640 1 librados: starting objecter 2024-03-29T11:44:25.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.608+0000 7fc3c67c7640 1 Processor -- start 2024-03-29T11:44:25.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.608+0000 7fc3c67c7640 1 -- start start 2024-03-29T11:44:25.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.608+0000 7fc3c67c7640 1 librados: setting wanted keys 2024-03-29T11:44:25.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.608+0000 7fc3c67c7640 1 librados: calling monclient init 2024-03-29T11:44:25.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.608+0000 7fc3c67c7640 10 monclient: init 2024-03-29T11:44:25.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.609+0000 7fc3c67c7640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:25.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.609+0000 7fc3c67c7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:25.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.610+0000 7fc3c67c7640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c00b1f60 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:25.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.611+0000 7fc3c67c7640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c0143d10 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:25.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.611+0000 7fc3c67c7640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c0153730 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:25.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.611+0000 7fc3c67c7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00a96f0 con 0x7fc3c0153730 2024-03-29T11:44:25.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.611+0000 7fc3c67c7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c01e4230 con 0x7fc3c00b1f60 2024-03-29T11:44:25.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.611+0000 7fc3c67c7640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c0154a30 con 0x7fc3c0143d10 2024-03-29T11:44:25.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.611+0000 7fc3c67c7640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:25.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.611+0000 7fc3c67c7640 10 monclient(hunting): authenticate will time out at 1078.890691s 2024-03-29T11:44:25.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc3c00b1f60 0x7fc3c0154320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:54771/0 (socket says 172.21.15.55:54771) 2024-03-29T11:44:25.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bffff640 1 -- 172.21.15.55:0/2030053276 learned_addr learned my addr 172.21.15.55:0/2030053276 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bcff9640 10 client.?.objecter ms_handle_connect 0x7fc3c00b1f60 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247815207 0 0) 0x7fc3c01e4230 con 0x7fc3c00b1f60 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bcff9640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.615+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac004eb0 con 0x7fc3c00b1f60 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.616+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1880850430 0 0) 0x7fc3ac004eb0 con 0x7fc3c00b1f60 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.616+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c01e4230 con 0x7fc3c00b1f60 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.616+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc3b8001ce0 con 0x7fc3c00b1f60 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.616+0000 7fc3bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.616+0000 7fc3bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:25.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.616+0000 7fc3bcff9640 10 monclient(hunting): dump: 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.616+0000 7fc3bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.616+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3726975404 0 0) 0x7fc3c01e4230 con 0x7fc3c00b1f60 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.616+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 >> v1:172.21.15.55:6790/0 conn(0x7fc3c0143d10 legacy=0x7fc3c01e4750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 >> v1:172.21.15.55:6789/0 conn(0x7fc3c0153730 legacy=0x7fc3c013dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:25.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 1 monclient: found mon.b 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c01e4e60 con 0x7fc3c00b1f60 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 10 client.?.objecter ms_handle_connect 0x7fc3c0153730 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3c67c7640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3c67c7640 10 monclient: _renew_subs 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3c67c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3c67c7640 1 -- 172.21.15.55:0/2030053276 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c013e440 con 0x7fc3c00b1f60 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3b8002000 con 0x7fc3c00b1f60 2024-03-29T11:44:25.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc3b8003dc0 con 0x7fc3c00b1f60 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.617+0000 7fc3bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.618+0000 7fc3bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.618+0000 7fc3bcff9640 10 monclient: dump: 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:25.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.618+0000 7fc3c67c7640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:25.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.618+0000 7fc3c67c7640 10 monclient: _renew_subs 2024-03-29T11:44:25.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.618+0000 7fc3c67c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:25.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.618+0000 7fc3c67c7640 1 -- 172.21.15.55:0/2030053276 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c006f800 con 0x7fc3c00b1f60 2024-03-29T11:44:25.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.618+0000 7fc3be7fc640 4 set_mon_vals no callback set 2024-03-29T11:44:25.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.618+0000 7fc3c67c7640 1 librados: init done 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.620+0000 7fc3a67fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.620+0000 7fc3a67fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.620+0000 7fc3a67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.620+0000 7fc3a67fc640 1 -- 172.21.15.55:0/2030053276 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c01e4090 con 0x7fc3c00b1f60 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.620+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fc3b8003f20 con 0x7fc3c00b1f60 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.621+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fc3b8084e80 con 0x7fc3c00b1f60 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.621+0000 7fc3bcff9640 10 client.4312.objecter ms_dispatch 0x7fc3c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.621+0000 7fc3bcff9640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.621+0000 7fc3bcff9640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.621+0000 7fc3bcff9640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.622+0000 7fc3bf7fe640 10 monclient: get_auth_request con 0x7fc3ac06dd40 auth_method 0 2024-03-29T11:44:25.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.623+0000 7fc3bcff9640 10 client.4312.objecter ms_handle_connect 0x7fc3ac06dd40 2024-03-29T11:44:25.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.628+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3c01e4090 con 0x7fc3c00b1f60 2024-03-29T11:44:25.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.629+0000 7fc3bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:25.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.629+0000 7fc3bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:25.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.968+0000 7fc3a67fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:25.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.968+0000 7fc3a67fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:25.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.968+0000 7fc3a67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:25.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.968+0000 7fc3a67fc640 1 -- 172.21.15.55:0/2030053276 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3c01e4090 con 0x7fc3c00b1f60 2024-03-29T11:44:25.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.973+0000 7fc3bcff9640 1 -- 172.21.15.55:0/2030053276 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fc3c01e4090 con 0x7fc3c00b1f60 2024-03-29T11:44:25.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.973+0000 7fc3bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:25.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.973+0000 7fc3bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:25.980 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:44:25.981 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:25.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.976+0000 7fc3a67fc640 10 librados: watch_flush enter 2024-03-29T11:44:25.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.977+0000 7fc3a67fc640 10 librados: watch_flush exit 2024-03-29T11:44:25.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.977+0000 7fc3a67fc640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:44:25.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.977+0000 7fc3a67fc640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:44:25.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.978+0000 7fc3a67fc640 1 -- 172.21.15.55:0/2030053276 >> v1:172.21.15.171:6816/1463197163 conn(0x7fc3ac06dd40 legacy=0x7fc3ac070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:25.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.978+0000 7fc3a67fc640 10 monclient: shutdown 2024-03-29T11:44:25.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.978+0000 7fc3a67fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:25.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.978+0000 7fc3a67fc640 1 -- 172.21.15.55:0/2030053276 >> v1:172.21.15.171:6789/0 conn(0x7fc3c00b1f60 legacy=0x7fc3c0154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:25.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.978+0000 7fc3a67fc640 1 -- 172.21.15.55:0/2030053276 shutdown_connections 2024-03-29T11:44:25.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.978+0000 7fc3a67fc640 1 -- 172.21.15.55:0/2030053276 >> 172.21.15.55:0/2030053276 conn(0x7fc3c0066a00 msgr2=0x7fc3c00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:25.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.978+0000 7fc3a67fc640 1 -- 172.21.15.55:0/2030053276 shutdown_connections 2024-03-29T11:44:25.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.979+0000 7fc3a67fc640 1 -- 172.21.15.55:0/2030053276 wait complete. 2024-03-29T11:44:25.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:25.979+0000 7fc3a67fc640 1 librados: shutdown 2024-03-29T11:44:26.001 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:32.808 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:32.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.960+0000 7f0928f96640 10 monclient: get_monmap_and_config 2024-03-29T11:44:32.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.960+0000 7f0928f96640 10 monclient: build_initial_monmap 2024-03-29T11:44:32.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.960+0000 7f0928f96640 10 monclient: monmap: 2024-03-29T11:44:32.964 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:32.964 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:32.964 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:44:32.961302+0000 2024-03-29T11:44:32.964 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:44:32.961302+0000 2024-03-29T11:44:32.965 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:32.965 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:32.965 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:32.965 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:44:32.965 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:44:32.965 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:32.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.961+0000 7f0928f96640 1 Processor -- start 2024-03-29T11:44:32.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.961+0000 7f0928f96640 1 -- start start 2024-03-29T11:44:32.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.961+0000 7f0928f96640 10 monclient: init 2024-03-29T11:44:32.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 10 monclient(hunting): picked mon.noname-c con 0x7f09241457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 10 monclient(hunting): picked mon.noname-a con 0x7f092413f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 10 monclient(hunting): picked mon.noname-b con 0x7f09240a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09240aaef0 con 0x7f092413f650 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0924145bd0 con 0x7f09241457f0 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0924145eb0 con 0x7f09240a9d70 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f0928f96640 10 monclient(hunting): authenticate will time out at 1086.242431s 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f092259c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f09241457f0 0x7f092413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:41550/0 (socket says 172.21.15.55:41550) 2024-03-29T11:44:32.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.962+0000 7f092259c640 1 -- 172.21.15.55:0/3091002173 learned_addr learned my addr 172.21.15.55:0/3091002173 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:32.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.963+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584730487 0 0) 0x7f09240aaef0 con 0x7f092413f650 2024-03-29T11:44:32.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.963+0000 7f092159a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:32.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.963+0000 7f092159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:32.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.963+0000 7f092159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:32.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.963+0000 7f092159a640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:44:32.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.963+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0910003520 con 0x7f092413f650 2024-03-29T11:44:32.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382617115 0 0) 0x7f0924145eb0 con 0x7f09240a9d70 2024-03-29T11:44:32.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:32.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:32.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:32.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09240aaef0 con 0x7f09240a9d70 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2583316660 0 0) 0x7f0910003520 con 0x7f092413f650 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 >> v1:172.21.15.55:6790/0 conn(0x7f09240a9d70 legacy=0x7f092414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 >> v1:172.21.15.171:6789/0 conn(0x7f09241457f0 legacy=0x7f092413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: _finish_hunting 0 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 monclient: found mon.noname-a 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924146340 con 0x7f092413f650 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: _finish_auth 0 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: _check_auth_tickets 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f090c001690 con 0x7f092413f650 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:32.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: dump: 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:32.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f0928f96640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f0928f96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f090c001870 con 0x7f092413f650 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 1 -- 172.21.15.55:0/3091002173 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f090c001690 con 0x7f092413f650 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f0928f96640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f0928f96640 4 set_mon_vals no callback set 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.964+0000 7f092159a640 10 monclient: dump: 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:32.973 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:32.974 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:32.974 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:32.974 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:32.974 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:32.974 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:32.974 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.965+0000 7f0928f96640 10 monclient: shutdown 2024-03-29T11:44:32.974 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.965+0000 7f0928f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:32.974 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.965+0000 7f0928f96640 1 -- 172.21.15.55:0/3091002173 >> v1:172.21.15.55:6789/0 conn(0x7f092413f650 legacy=0x7f09240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:32.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.965+0000 7f0928f96640 1 -- 172.21.15.55:0/3091002173 shutdown_connections 2024-03-29T11:44:32.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.965+0000 7f0928f96640 1 -- 172.21.15.55:0/3091002173 >> 172.21.15.55:0/3091002173 conn(0x7f0924066a00 msgr2=0x7f0924066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:32.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.965+0000 7f0928f96640 1 -- 172.21.15.55:0/3091002173 shutdown_connections 2024-03-29T11:44:32.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.965+0000 7f0928f96640 1 -- 172.21.15.55:0/3091002173 wait complete. 2024-03-29T11:44:32.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.967+0000 7f0928f96640 10 monclient: build_initial_monmap 2024-03-29T11:44:32.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.967+0000 7f0928f96640 10 monclient: monmap: 2024-03-29T11:44:32.981 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:32.981 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:32.981 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.968+0000 7f0928f96640 1 librados: starting msgr at 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.968+0000 7f0928f96640 1 librados: starting objecter 2024-03-29T11:44:32.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.968+0000 7f0928f96640 1 Processor -- start 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.968+0000 7f0928f96640 1 -- start start 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.968+0000 7f0928f96640 1 librados: setting wanted keys 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.968+0000 7f0928f96640 1 librados: calling monclient init 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.968+0000 7f0928f96640 10 monclient: init 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.969+0000 7f0928f96640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.969+0000 7f0928f96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.969+0000 7f0928f96640 10 monclient(hunting): picked mon.noname-b con 0x7f09240a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.969+0000 7f0928f96640 10 monclient(hunting): picked mon.noname-a con 0x7f09241457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f0921d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f09241457f0 0x7f09241dc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:38642/0 (socket says 172.21.15.55:38642) 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f0921d9b640 1 -- 172.21.15.55:0/3319957291 learned_addr learned my addr 172.21.15.55:0/3319957291 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f0928f96640 10 monclient(hunting): picked mon.noname-c con 0x7f092406e810 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09241dbb90 con 0x7f09241457f0 2024-03-29T11:44:32.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 client.?.objecter ms_handle_connect 0x7f09241457f0 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 client.?.objecter ms_handle_connect 0x7f092406e810 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0924138da0 con 0x7f09240a9d70 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092406f750 con 0x7f092406e810 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f0928f96640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f0928f96640 10 monclient(hunting): authenticate will time out at 1086.249994s 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140498360 0 0) 0x7f092406f750 con 0x7f092406e810 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:32.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0904004eb0 con 0x7f092406e810 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663355672 0 0) 0x7f09241dbb90 con 0x7f09241457f0 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.970+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f092406f750 con 0x7f09241457f0 2024-03-29T11:44:32.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 784769470 0 0) 0x7f0904004eb0 con 0x7f092406e810 2024-03-29T11:44:32.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09241dbb90 con 0x7f092406e810 2024-03-29T11:44:32.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0914001930 con 0x7f092406e810 2024-03-29T11:44:32.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:32.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:32.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 10 monclient(hunting): dump: 2024-03-29T11:44:32.986 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:32.986 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:32.986 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 10 client.?.objecter ms_handle_connect 0x7f09240a9d70 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.971+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923776127 0 0) 0x7f0924138da0 con 0x7f09240a9d70 2024-03-29T11:44:32.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:32.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:32.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:32.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:44:32.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0904004eb0 con 0x7f09240a9d70 2024-03-29T11:44:32.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 32399450 0 0) 0x7f09241dbb90 con 0x7f092406e810 2024-03-29T11:44:32.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 >> v1:172.21.15.171:6789/0 conn(0x7f09240a9d70 legacy=0x7f092406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:32.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 >> v1:172.21.15.55:6789/0 conn(0x7f09241457f0 legacy=0x7f09241dc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:32.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 1 monclient: found mon.c 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092406fa30 con 0x7f092406e810 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 10 monclient: _finish_auth 0 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f091affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f0928f96640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f0928f96640 10 monclient: _renew_subs 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f0928f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09241dc730 con 0x7f092406e810 2024-03-29T11:44:32.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f0928f96640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f0928f96640 10 monclient: _renew_subs 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f0928f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09241e7dd0 con 0x7f092406e810 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.972+0000 7f0928f96640 1 librados: init done 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.973+0000 7f0928f96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.973+0000 7f0928f96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.973+0000 7f0928f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.973+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092406ece0 con 0x7f092406e810 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.974+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0914003610 con 0x7f092406e810 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.974+0000 7f091affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:32.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.974+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0914003a20 con 0x7f092406e810 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.974+0000 7f091affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.974+0000 7f091affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.974+0000 7f091affd640 10 monclient: dump: 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:32.991 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.974+0000 7f0920d99640 4 set_mon_vals no callback set 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.974+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f0914003c40 con 0x7f092406e810 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.975+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f09140837d0 con 0x7f092406e810 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.975+0000 7f091affd640 10 client.4325.objecter ms_dispatch 0x7f0924000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.975+0000 7f091affd640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.975+0000 7f091affd640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.975+0000 7f091affd640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.975+0000 7f092259c640 10 monclient: get_auth_request con 0x7f090406de50 auth_method 0 2024-03-29T11:44:32.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.976+0000 7f091affd640 10 client.4325.objecter ms_handle_connect 0x7f090406de50 2024-03-29T11:44:32.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.985+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f091404f0b0 con 0x7f092406e810 2024-03-29T11:44:32.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.985+0000 7f091affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:32.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:32.985+0000 7f091affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:33.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.377+0000 7f0928f96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:33.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.377+0000 7f0928f96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:33.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.377+0000 7f0928f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:33.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.377+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09240b76c0 con 0x7f092406e810 2024-03-29T11:44:33.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.378+0000 7f091affd640 1 -- 172.21.15.55:0/3319957291 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f0914050fe0 con 0x7f092406e810 2024-03-29T11:44:33.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.378+0000 7f091affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:33.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.378+0000 7f091affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:33.383 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:44:33.383 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:33.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 10 librados: watch_flush enter 2024-03-29T11:44:33.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 10 librados: watch_flush exit 2024-03-29T11:44:33.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T11:44:33.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 10 client.4325.objecter successfully canceled tick 2024-03-29T11:44:33.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 >> v1:172.21.15.171:6816/1463197163 conn(0x7f090406de50 legacy=0x7f0904070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:33.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 10 monclient: shutdown 2024-03-29T11:44:33.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:33.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 >> v1:172.21.15.55:6790/0 conn(0x7f092406e810 legacy=0x7f09241e66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:33.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 shutdown_connections 2024-03-29T11:44:33.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 >> 172.21.15.55:0/3319957291 conn(0x7f0924066a00 msgr2=0x7f092413e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:33.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.382+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 shutdown_connections 2024-03-29T11:44:33.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.383+0000 7f0928f96640 1 -- 172.21.15.55:0/3319957291 wait complete. 2024-03-29T11:44:33.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:33.383+0000 7f0928f96640 1 librados: shutdown 2024-03-29T11:44:33.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:40.209 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.346+0000 7f31bb59e640 10 monclient: get_monmap_and_config 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.347+0000 7f31bb59e640 10 monclient: build_initial_monmap 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.347+0000 7f31bb59e640 10 monclient: monmap: 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:44:40.348193+0000 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:44:40.348193+0000 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:40.351 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:44:40.352 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:44:40.352 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:40.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.347+0000 7f31bb59e640 1 Processor -- start 2024-03-29T11:44:40.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.347+0000 7f31bb59e640 1 -- start start 2024-03-29T11:44:40.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.347+0000 7f31bb59e640 10 monclient: init 2024-03-29T11:44:40.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.349+0000 7f31bb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:40.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.349+0000 7f31bb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:40.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.349+0000 7f31bb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc0b2800 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:40.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.349+0000 7f31bb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f31bc0a9c00 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:40.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.349+0000 7f31b9d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f31bc0a9c00 0x7f31bc13bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:34500/0 (socket says 172.21.15.55:34500) 2024-03-29T11:44:40.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b9d9b640 1 -- 172.21.15.55:0/258122165 learned_addr learned my addr 172.21.15.55:0/258122165 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:40.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31bb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f31bc0b38c0 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:40.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31bb59e640 1 -- 172.21.15.55:0/258122165 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc141eb0 con 0x7f31bc0b2800 2024-03-29T11:44:40.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31bb59e640 1 -- 172.21.15.55:0/258122165 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc0b22d0 con 0x7f31bc0a9c00 2024-03-29T11:44:40.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31bb59e640 1 -- 172.21.15.55:0/258122165 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc142d30 con 0x7f31bc0b38c0 2024-03-29T11:44:40.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31bb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:40.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31bb59e640 10 monclient(hunting): authenticate will time out at 1093.631158s 2024-03-29T11:44:40.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517366987 0 0) 0x7f31bc0b22d0 con 0x7f31bc0a9c00 2024-03-29T11:44:40.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:40.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:40.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:40.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:44:40.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a4003520 con 0x7f31bc0a9c00 2024-03-29T11:44:40.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795046401 0 0) 0x7f31bc142d30 con 0x7f31bc0b38c0 2024-03-29T11:44:40.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:40.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:40.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:40.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:44:40.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.351+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31bc0b22d0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3617641279 0 0) 0x7f31a4003520 con 0x7f31bc0a9c00 2024-03-29T11:44:40.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 >> v1:172.21.15.55:6790/0 conn(0x7f31bc0b38c0 legacy=0x7f31bc14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 >> v1:172.21.15.55:6789/0 conn(0x7f31bc0b2800 legacy=0x7f31bc142620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 10 monclient: _finish_hunting 0 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 1 monclient: found mon.noname-c 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0aaf90 con 0x7f31bc0a9c00 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 10 monclient: _finish_auth 0 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 10 monclient: _check_auth_tickets 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f31b0001690 con 0x7f31bc0a9c00 2024-03-29T11:44:40.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31bb59e640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.352+0000 7f31bb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.353+0000 7f31b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.353+0000 7f31b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.353+0000 7f31b959a640 10 monclient: dump: 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:40.373 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:40.374 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:40.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.353+0000 7f31bb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:40.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.353+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31b0001870 con 0x7f31bc0a9c00 2024-03-29T11:44:40.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.353+0000 7f31b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:40.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.353+0000 7f31bb59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:40.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.353+0000 7f31bb59e640 4 set_mon_vals no callback set 2024-03-29T11:44:40.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.353+0000 7f31b959a640 1 -- 172.21.15.55:0/258122165 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f31b0001c00 con 0x7f31bc0a9c00 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31b959a640 10 monclient: dump: 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:40.375 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31bb59e640 10 monclient: shutdown 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31bb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31bb59e640 1 -- 172.21.15.55:0/258122165 >> v1:172.21.15.171:6789/0 conn(0x7f31bc0a9c00 legacy=0x7f31bc13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31bb59e640 1 -- 172.21.15.55:0/258122165 shutdown_connections 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31bb59e640 1 -- 172.21.15.55:0/258122165 >> 172.21.15.55:0/258122165 conn(0x7f31bc066a00 msgr2=0x7f31bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31bb59e640 1 -- 172.21.15.55:0/258122165 shutdown_connections 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.354+0000 7f31bb59e640 1 -- 172.21.15.55:0/258122165 wait complete. 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.355+0000 7f31bb59e640 10 monclient: build_initial_monmap 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.355+0000 7f31bb59e640 10 monclient: monmap: 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:44:40.376 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.356+0000 7f31bb59e640 1 librados: starting msgr at 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.356+0000 7f31bb59e640 1 librados: starting objecter 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.356+0000 7f31bb59e640 1 Processor -- start 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.356+0000 7f31bb59e640 1 -- start start 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.356+0000 7f31bb59e640 1 librados: setting wanted keys 2024-03-29T11:44:40.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.356+0000 7f31bb59e640 1 librados: calling monclient init 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.356+0000 7f31bb59e640 10 monclient: init 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc0a9c00 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f31bc0b2800 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f31bc0b38c0 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc1da210 con 0x7f31bc0a9c00 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc0afa80 con 0x7f31bc0b2800 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc1dadb0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:40.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bb59e640 10 monclient(hunting): authenticate will time out at 1093.637251s 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bad9d640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f31bc0b38c0 0x7f31bc1e1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:36632/0 (socket says 172.21.15.55:36632) 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31bad9d640 1 -- 172.21.15.55:0/4021115723 learned_addr learned my addr 172.21.15.55:0/4021115723 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.357+0000 7f31aaffd640 10 client.?.objecter ms_handle_connect 0x7f31bc0b38c0 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974442000 0 0) 0x7f31bc1dadb0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f319c004eb0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2902602935 0 0) 0x7f319c004eb0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31bc1dadb0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f31b4001c70 con 0x7f31bc0b38c0 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 10 monclient(hunting): dump: 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:40.380 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:40.381 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:40.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:40.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2930017862 0 0) 0x7f31bc1dadb0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 >> v1:172.21.15.171:6789/0 conn(0x7f31bc0b2800 legacy=0x7f31bc0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:40.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.358+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 >> v1:172.21.15.55:6789/0 conn(0x7f31bc0a9c00 legacy=0x7f31bc1da6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:40.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.359+0000 7f31aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:40.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.359+0000 7f31aaffd640 1 monclient: found mon.c 2024-03-29T11:44:40.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.359+0000 7f31aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.359+0000 7f31aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.359+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc1daf40 con 0x7f31bc0b38c0 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.359+0000 7f31aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.359+0000 7f31aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.359+0000 7f31aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31bb59e640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31bb59e640 10 monclient: _renew_subs 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31bc0b01f0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31bb59e640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31bb59e640 10 monclient: _renew_subs 2024-03-29T11:44:40.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31bc0ae860 con 0x7f31bc0b38c0 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31bb59e640 1 librados: init done 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31b4003970 con 0x7f31bc0b38c0 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f31b4003d80 con 0x7f31bc0b38c0 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.360+0000 7f31aaffd640 10 monclient: dump: 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:40.383 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.361+0000 7f31abfff640 4 set_mon_vals no callback set 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.362+0000 7f31bb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.362+0000 7f31bb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.362+0000 7f31bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.362+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a0005080 con 0x7f31bc0b38c0 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.362+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f31b4003ee0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.363+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f31b4084d70 con 0x7f31bc0b38c0 2024-03-29T11:44:40.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.363+0000 7f31aaffd640 10 client.4331.objecter ms_dispatch 0x7f31bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:40.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.363+0000 7f31aaffd640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:40.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.363+0000 7f31aaffd640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:40.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.363+0000 7f31aaffd640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:44:40.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.363+0000 7f31ba59c640 10 monclient: get_auth_request con 0x7f319c06dba0 auth_method 0 2024-03-29T11:44:40.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.364+0000 7f31aaffd640 10 client.4331.objecter ms_handle_connect 0x7f319c06dba0 2024-03-29T11:44:40.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.370+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31b40b48b0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.370+0000 7f31aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:40.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.370+0000 7f31aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:40.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.691+0000 7f31bb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:40.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.691+0000 7f31bb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:40.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.691+0000 7f31bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:44:40.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.691+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31a0005330 con 0x7f31bc0b38c0 2024-03-29T11:44:40.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.691+0000 7f31aaffd640 1 -- 172.21.15.55:0/4021115723 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f31b40504e0 con 0x7f31bc0b38c0 2024-03-29T11:44:40.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.691+0000 7f31aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:40.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.691+0000 7f31aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:40.697 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:44:40.697 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.694+0000 7f31bb59e640 10 librados: watch_flush enter 2024-03-29T11:44:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.694+0000 7f31bb59e640 10 librados: watch_flush exit 2024-03-29T11:44:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.694+0000 7f31bb59e640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:44:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.694+0000 7f31bb59e640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:44:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.694+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 >> v1:172.21.15.171:6816/1463197163 conn(0x7f319c06dba0 legacy=0x7f319c070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.694+0000 7f31bb59e640 10 monclient: shutdown 2024-03-29T11:44:40.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.694+0000 7f31bb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:40.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.694+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 >> v1:172.21.15.55:6790/0 conn(0x7f31bc0b38c0 legacy=0x7f31bc1e1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:40.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.695+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 shutdown_connections 2024-03-29T11:44:40.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.695+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 >> 172.21.15.55:0/4021115723 conn(0x7f31bc066a00 msgr2=0x7f31bc145240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:40.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.695+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 shutdown_connections 2024-03-29T11:44:40.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.695+0000 7f31bb59e640 1 -- 172.21.15.55:0/4021115723 wait complete. 2024-03-29T11:44:40.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:40.695+0000 7f31bb59e640 1 librados: shutdown 2024-03-29T11:44:40.715 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:47.522 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:47.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.672+0000 7f6372664640 10 monclient: get_monmap_and_config 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.672+0000 7f6372664640 10 monclient: build_initial_monmap 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.672+0000 7f6372664640 10 monclient: monmap: 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:44:47.672820+0000 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:44:47.672820+0000 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:44:47.676 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 1 Processor -- start 2024-03-29T11:44:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 1 -- start start 2024-03-29T11:44:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 10 monclient: init 2024-03-29T11:44:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 10 monclient(hunting): picked mon.noname-a con 0x7f636c0b0500 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 10 monclient(hunting): picked mon.noname-b con 0x7f636c0b08e0 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 10 monclient(hunting): picked mon.noname-c con 0x7f636c0aeca0 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c140620 con 0x7f636c0b0500 2024-03-29T11:44:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c13daf0 con 0x7f636c0aeca0 2024-03-29T11:44:47.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c0ae180 con 0x7f636c0b08e0 2024-03-29T11:44:47.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.674+0000 7f6372664640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:47.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.675+0000 7f6372664640 10 monclient(hunting): authenticate will time out at 1100.953652s 2024-03-29T11:44:47.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.676+0000 7f6370c01640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f636c0aeca0 0x7f636c14b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:54946/0 (socket says 172.21.15.55:54946) 2024-03-29T11:44:47.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.676+0000 7f6370c01640 1 -- 172.21.15.55:0/3335222520 learned_addr learned my addr 172.21.15.55:0/3335222520 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.677+0000 7f636affd640 1 -- 172.21.15.55:0/3335222520 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21585099 0 0) 0x7f636c13daf0 con 0x7f636c0aeca0 2024-03-29T11:44:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.677+0000 7f636affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.677+0000 7f636affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.677+0000 7f636affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.677+0000 7f636affd640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:44:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.677+0000 7f636affd640 1 -- 172.21.15.55:0/3335222520 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6354003520 con 0x7f636c0aeca0 2024-03-29T11:44:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.677+0000 7f636affd640 1 -- 172.21.15.55:0/3335222520 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1773331572 0 0) 0x7f6354003520 con 0x7f636c0aeca0 2024-03-29T11:44:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.677+0000 7f636affd640 1 -- 172.21.15.55:0/3335222520 >> v1:172.21.15.55:6790/0 conn(0x7f636c0b08e0 legacy=0x7f636c13d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 1 -- 172.21.15.55:0/3335222520 >> v1:172.21.15.55:6789/0 conn(0x7f636c0b0500 legacy=0x7f636c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 1 monclient: found mon.noname-c 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 1 -- 172.21.15.55:0/3335222520 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c072510 con 0x7f636c0aeca0 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: _finish_auth 0 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f6372664640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f6372664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 1 -- 172.21.15.55:0/3335222520 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6364001a10 con 0x7f636c0aeca0 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: dump: 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:47.684 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f6372664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 1 -- 172.21.15.55:0/3335222520 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6364001ca0 con 0x7f636c0aeca0 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f6372664640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 1 -- 172.21.15.55:0/3335222520 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6364002030 con 0x7f636c0aeca0 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f6372664640 4 set_mon_vals no callback set 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.679+0000 7f636affd640 10 monclient: dump: 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:47.685 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:47.686 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:47.686 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:47.686 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:47.686 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:47.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.681+0000 7f6372664640 10 monclient: shutdown 2024-03-29T11:44:47.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.681+0000 7f6372664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:47.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.681+0000 7f6372664640 1 -- 172.21.15.55:0/3335222520 >> v1:172.21.15.171:6789/0 conn(0x7f636c0aeca0 legacy=0x7f636c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:47.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.681+0000 7f6372664640 1 -- 172.21.15.55:0/3335222520 shutdown_connections 2024-03-29T11:44:47.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.681+0000 7f6372664640 1 -- 172.21.15.55:0/3335222520 >> 172.21.15.55:0/3335222520 conn(0x7f636c06b660 msgr2=0x7f636c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.682+0000 7f6372664640 1 -- 172.21.15.55:0/3335222520 shutdown_connections 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.682+0000 7f6372664640 1 -- 172.21.15.55:0/3335222520 wait complete. 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.683+0000 7f6372664640 10 monclient: build_initial_monmap 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.683+0000 7f6372664640 10 monclient: monmap: 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:47.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.683+0000 7f6372664640 1 librados: starting msgr at 2024-03-29T11:44:47.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.683+0000 7f6372664640 1 librados: starting objecter 2024-03-29T11:44:47.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.684+0000 7f6372664640 1 Processor -- start 2024-03-29T11:44:47.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.684+0000 7f6372664640 1 -- start start 2024-03-29T11:44:47.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.684+0000 7f6372664640 1 librados: setting wanted keys 2024-03-29T11:44:47.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.684+0000 7f6372664640 1 librados: calling monclient init 2024-03-29T11:44:47.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.684+0000 7f6372664640 10 monclient: init 2024-03-29T11:44:47.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.685+0000 7f6372664640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:47.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.685+0000 7f6372664640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:47.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.685+0000 7f6372664640 10 monclient(hunting): picked mon.noname-b con 0x7f636c0aeca0 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:47.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.685+0000 7f6372664640 10 monclient(hunting): picked mon.noname-c con 0x7f636c0b0500 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:47.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.688+0000 7f636bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f636c0aeca0 0x7f636c12f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:55334/0 (socket says 172.21.15.55:55334) 2024-03-29T11:44:47.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.688+0000 7f6370c01640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f636c12ce10 0x7f636c1e27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:38953/0 (socket says 172.21.15.55:38953) 2024-03-29T11:44:47.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.688+0000 7f636bfff640 1 -- 172.21.15.55:0/1036484742 learned_addr learned my addr 172.21.15.55:0/1036484742 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:47.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.688+0000 7f6372664640 10 monclient(hunting): picked mon.noname-a con 0x7f636c12ce10 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:47.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c12f110 con 0x7f636c12ce10 2024-03-29T11:44:47.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c12f380 con 0x7f636c0aeca0 2024-03-29T11:44:47.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c06e030 con 0x7f636c0b0500 2024-03-29T11:44:47.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6372664640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:47.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6372664640 10 monclient(hunting): authenticate will time out at 1100.968139s 2024-03-29T11:44:47.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6368ff9640 10 client.?.objecter ms_handle_connect 0x7f636c0aeca0 2024-03-29T11:44:47.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6368ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:47.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6368ff9640 10 client.?.objecter ms_handle_connect 0x7f636c12ce10 2024-03-29T11:44:47.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6368ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:47.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.689+0000 7f6368ff9640 10 client.?.objecter ms_handle_connect 0x7f636c0b0500 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337399238 0 0) 0x7f636c12f380 con 0x7f636c0aeca0 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6358004eb0 con 0x7f636c0aeca0 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999059374 0 0) 0x7f636c12f110 con 0x7f636c12ce10 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f636c12f380 con 0x7f636c12ce10 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473478944 0 0) 0x7f636c06e030 con 0x7f636c0b0500 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.690+0000 7f6368ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f636c12f110 con 0x7f636c0b0500 2024-03-29T11:44:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2250816827 0 0) 0x7f636c12f380 con 0x7f636c12ce10 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f636c06e030 con 0x7f636c12ce10 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f63640018d0 con 0x7f636c12ce10 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:47.702 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3447297789 0 0) 0x7f636c06e030 con 0x7f636c12ce10 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 >> v1:172.21.15.55:6790/0 conn(0x7f636c0b0500 legacy=0x7f636c1df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.691+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 >> v1:172.21.15.171:6789/0 conn(0x7f636c0aeca0 legacy=0x7f636c12f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6368ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6368ff9640 1 monclient: found mon.a 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6368ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6368ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c12d3e0 con 0x7f636c12ce10 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6368ff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6368ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6368ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6372664640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:44:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6372664640 10 monclient: _renew_subs 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6372664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c070890 con 0x7f636c12ce10 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6372664640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6372664640 10 monclient: _renew_subs 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6372664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c070dc0 con 0x7f636c12ce10 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.692+0000 7f6372664640 1 librados: init done 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.693+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6364001a30 con 0x7f636c12ce10 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.693+0000 7f6368ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.693+0000 7f636a7fc640 4 set_mon_vals no callback set 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6372664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6364005110 con 0x7f636c12ce10 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6368ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6368ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6368ff9640 10 monclient: dump: 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f636401b0e0 con 0x7f636c12ce10 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6372664640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6372664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f6364084cd0 con 0x7f636c12ce10 2024-03-29T11:44:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.695+0000 7f6368ff9640 10 client.4317.objecter ms_dispatch 0x7f636c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:47.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.696+0000 7f6368ff9640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:47.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.696+0000 7f6368ff9640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:47.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.696+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634c005080 con 0x7f636c12ce10 2024-03-29T11:44:47.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.696+0000 7f6368ff9640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T11:44:47.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.696+0000 7f636bfff640 10 monclient: get_auth_request con 0x7f635806dce0 auth_method 0 2024-03-29T11:44:47.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.697+0000 7f6368ff9640 10 client.4317.objecter ms_handle_connect 0x7f635806dce0 2024-03-29T11:44:47.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.705+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6364050630 con 0x7f636c12ce10 2024-03-29T11:44:47.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.705+0000 7f6368ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:47.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:47.705+0000 7f6368ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:48.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.032+0000 7f6372664640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:48.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.032+0000 7f6372664640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:48.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.032+0000 7f6372664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:44:48.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.032+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f634c005330 con 0x7f636c12ce10 2024-03-29T11:44:48.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.033+0000 7f6368ff9640 1 -- 172.21.15.55:0/1036484742 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f6364052560 con 0x7f636c12ce10 2024-03-29T11:44:48.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.033+0000 7f6368ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:48.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.033+0000 7f6368ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:48.037 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:44:48.037 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:48.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.036+0000 7f6372664640 10 librados: watch_flush enter 2024-03-29T11:44:48.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 10 librados: watch_flush exit 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 10 client.4317.objecter successfully canceled tick 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 >> v1:172.21.15.171:6816/1463197163 conn(0x7f635806dce0 legacy=0x7f63580701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 10 monclient: shutdown 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 >> v1:172.21.15.55:6789/0 conn(0x7f636c12ce10 legacy=0x7f636c1e27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 shutdown_connections 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 >> 172.21.15.55:0/1036484742 conn(0x7f636c06b660 msgr2=0x7f636c0a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 shutdown_connections 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.037+0000 7f6372664640 1 -- 172.21.15.55:0/1036484742 wait complete. 2024-03-29T11:44:48.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:48.038+0000 7f6372664640 1 librados: shutdown 2024-03-29T11:44:48.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:54.867 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:55.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.015+0000 7fc8ae47b640 10 monclient: get_monmap_and_config 2024-03-29T11:44:55.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.015+0000 7fc8ae47b640 10 monclient: build_initial_monmap 2024-03-29T11:44:55.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.015+0000 7fc8ae47b640 10 monclient: monmap: 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:44:55.017131+0000 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:44:55.017131+0000 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:55.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.018+0000 7fc8ae47b640 1 Processor -- start 2024-03-29T11:44:55.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.018+0000 7fc8ae47b640 1 -- start start 2024-03-29T11:44:55.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.018+0000 7fc8ae47b640 10 monclient: init 2024-03-29T11:44:55.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:55.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:55.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8a80b0770 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:55.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 10 monclient(hunting): picked mon.noname-b con 0x7fc8a814a700 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:55.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 10 monclient(hunting): picked mon.noname-c con 0x7fc8a80ac5a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:55.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a80aba70 con 0x7fc8a80b0770 2024-03-29T11:44:55.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a80aecb0 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a80af150 con 0x7fc8a814a700 2024-03-29T11:44:55.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:55.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.019+0000 7fc8ae47b640 10 monclient(hunting): authenticate will time out at 1108.299065s 2024-03-29T11:44:55.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.024+0000 7fc8a7fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc8a80b0770 0x7fc8a8149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46175/0 (socket says 172.21.15.55:46175) 2024-03-29T11:44:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.024+0000 7fc8a7fff640 1 -- 172.21.15.55:0/2652716870 learned_addr learned my addr 172.21.15.55:0/2652716870 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.024+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465669501 0 0) 0x7fc8a80aba70 con 0x7fc8a80b0770 2024-03-29T11:44:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:44:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc890003520 con 0x7fc8a80b0770 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640456808 0 0) 0x7fc8a80af150 con 0x7fc8a814a700 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a80aba70 con 0x7fc8a814a700 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178922547 0 0) 0x7fc890003520 con 0x7fc8a80b0770 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 >> v1:172.21.15.55:6790/0 conn(0x7fc8a814a700 legacy=0x7fc8a80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 >> v1:172.21.15.171:6789/0 conn(0x7fc8a80ac5a0 legacy=0x7fc8a8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:44:55.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.025+0000 7fc8a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a80ac9d0 con 0x7fc8a80b0770 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc89c001690 con 0x7fc8a80b0770 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: dump: 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:55.034 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:55.035 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:55.035 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:55.035 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:55.035 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:55.035 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:55.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8ae47b640 5 monclient: authenticate success, global_id 4320 2024-03-29T11:44:55.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8ae47b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:55.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc89c001870 con 0x7fc8a80b0770 2024-03-29T11:44:55.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 1 -- 172.21.15.55:0/2652716870 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc89c001c80 con 0x7fc8a80b0770 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8a6ffd640 10 monclient: dump: 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:55.036 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:55.037 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:55.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8ae47b640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:55.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.026+0000 7fc8ae47b640 4 set_mon_vals no callback set 2024-03-29T11:44:55.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.027+0000 7fc8ae47b640 10 monclient: shutdown 2024-03-29T11:44:55.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.027+0000 7fc8ae47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:55.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.027+0000 7fc8ae47b640 1 -- 172.21.15.55:0/2652716870 >> v1:172.21.15.55:6789/0 conn(0x7fc8a80b0770 legacy=0x7fc8a8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.029+0000 7fc8ae47b640 1 -- 172.21.15.55:0/2652716870 shutdown_connections 2024-03-29T11:44:55.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.029+0000 7fc8ae47b640 1 -- 172.21.15.55:0/2652716870 >> 172.21.15.55:0/2652716870 conn(0x7fc8a806b660 msgr2=0x7fc8a806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:55.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.030+0000 7fc8ae47b640 1 -- 172.21.15.55:0/2652716870 shutdown_connections 2024-03-29T11:44:55.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.030+0000 7fc8ae47b640 1 -- 172.21.15.55:0/2652716870 wait complete. 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.031+0000 7fc8ae47b640 10 monclient: build_initial_monmap 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.031+0000 7fc8ae47b640 10 monclient: monmap: 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.031+0000 7fc8ae47b640 1 librados: starting msgr at 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.031+0000 7fc8ae47b640 1 librados: starting objecter 2024-03-29T11:44:55.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.032+0000 7fc8ae47b640 1 Processor -- start 2024-03-29T11:44:55.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.032+0000 7fc8ae47b640 1 -- start start 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.032+0000 7fc8ae47b640 1 librados: setting wanted keys 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.032+0000 7fc8ae47b640 1 librados: calling monclient init 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.032+0000 7fc8ae47b640 10 monclient: init 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 10 monclient(hunting): picked mon.noname-b con 0x7fc8a80ac5a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 10 monclient(hunting): picked mon.noname-c con 0x7fc8a80b0770 addr v1:172.21.15.55:6790/0 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8a814a700 addr v1:172.21.15.55:6789/0 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a8143ca0 con 0x7fc8a814a700 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a806db00 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a8071230 con 0x7fc8a80b0770 2024-03-29T11:44:55.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:55.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.033+0000 7fc8ae47b640 10 monclient(hunting): authenticate will time out at 1108.313111s 2024-03-29T11:44:55.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.037+0000 7fc8a77fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fc8a80b0770 0x7fc8a8142e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:39191/0 (socket says 172.21.15.55:39191) 2024-03-29T11:44:55.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.038+0000 7fc8a77fe640 1 -- 172.21.15.55:0/4255741592 learned_addr learned my addr 172.21.15.55:0/4255741592 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 10 client.?.objecter ms_handle_connect 0x7fc8a80b0770 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 10 client.?.objecter ms_handle_connect 0x7fc8a80ac5a0 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104341919 0 0) 0x7fc8a806db00 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:44:55.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.040+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc894004eb0 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.041+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3430790948 0 0) 0x7fc894004eb0 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.041+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8a806db00 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.041+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc89c001b80 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.041+0000 7fc8a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.041+0000 7fc8a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.041+0000 7fc8a4ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:55.053 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.041+0000 7fc8a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.041+0000 7fc8a4ff9640 10 client.?.objecter ms_handle_connect 0x7fc8a814a700 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832675326 0 0) 0x7fc8a8143ca0 con 0x7fc8a814a700 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc894004eb0 con 0x7fc8a814a700 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886248849 0 0) 0x7fc8a8071230 con 0x7fc8a80b0770 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:55.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a8143ca0 con 0x7fc8a80b0770 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 922579308 0 0) 0x7fc8a806db00 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 >> v1:172.21.15.55:6790/0 conn(0x7fc8a80b0770 legacy=0x7fc8a8142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 >> v1:172.21.15.55:6789/0 conn(0x7fc8a814a700 legacy=0x7fc8a8143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 1 monclient: found mon.b 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a8071510 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8ae47b640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8ae47b640 10 monclient: _renew_subs 2024-03-29T11:44:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8ae47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a80717f0 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8ae47b640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8ae47b640 10 monclient: _renew_subs 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8ae47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a806e6f0 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.042+0000 7fc8ae47b640 1 librados: init done 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.044+0000 7fc8ae47b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.044+0000 7fc8ae47b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.044+0000 7fc8ae47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.044+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a80ab560 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.047+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc89c004bf0 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.047+0000 7fc8a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.047+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc89c004f00 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.047+0000 7fc8a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.047+0000 7fc8a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.047+0000 7fc8a4ff9640 10 monclient: dump: 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.047+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fc89c01aed0 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.047+0000 7fc8a67fc640 4 set_mon_vals no callback set 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.048+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fc89c084c70 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.048+0000 7fc8a4ff9640 10 client.4330.objecter ms_dispatch 0x7fc8a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.048+0000 7fc8a4ff9640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.048+0000 7fc8a4ff9640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.048+0000 7fc8a4ff9640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:44:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.048+0000 7fc8a77fe640 10 monclient: get_auth_request con 0x7fc89406dec0 auth_method 0 2024-03-29T11:44:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.049+0000 7fc8a4ff9640 10 client.4330.objecter ms_handle_connect 0x7fc89406dec0 2024-03-29T11:44:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.052+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc89c050400 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.052+0000 7fc8a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:55.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.052+0000 7fc8a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:55.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.384+0000 7fc8ae47b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:55.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.384+0000 7fc8ae47b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:55.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.384+0000 7fc8ae47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:44:55.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.384+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8a8075f10 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.385+0000 7fc8a4ff9640 1 -- 172.21.15.55:0/4255741592 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fc89c052500 con 0x7fc8a80ac5a0 2024-03-29T11:44:55.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.385+0000 7fc8a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:55.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.385+0000 7fc8a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:55.393 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:44:55.393 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.388+0000 7fc8ae47b640 10 librados: watch_flush enter 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 10 librados: watch_flush exit 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 >> v1:172.21.15.171:6816/1463197163 conn(0x7fc89406dec0 legacy=0x7fc894070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 10 monclient: shutdown 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 >> v1:172.21.15.171:6789/0 conn(0x7fc8a80ac5a0 legacy=0x7fc8a806dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 shutdown_connections 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 >> 172.21.15.55:0/4255741592 conn(0x7fc8a806b660 msgr2=0x7fc8a8149560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:55.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 shutdown_connections 2024-03-29T11:44:55.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 1 -- 172.21.15.55:0/4255741592 wait complete. 2024-03-29T11:44:55.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:44:55.389+0000 7fc8ae47b640 1 librados: shutdown 2024-03-29T11:44:55.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:02.221 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:02.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.368+0000 7fbc2f503640 10 monclient: get_monmap_and_config 2024-03-29T11:45:02.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.368+0000 7fbc2f503640 10 monclient: build_initial_monmap 2024-03-29T11:45:02.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.368+0000 7fbc2f503640 10 monclient: monmap: 2024-03-29T11:45:02.373 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:02.373 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:02.373 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:45:02.369200+0000 2024-03-29T11:45:02.373 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:45:02.369200+0000 2024-03-29T11:45:02.373 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:02.373 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:02.373 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:02.374 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:45:02.374 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:45:02.374 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:02.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.371+0000 7fbc2f503640 1 Processor -- start 2024-03-29T11:45:02.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.374+0000 7fbc2f503640 1 -- start start 2024-03-29T11:45:02.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.374+0000 7fbc2f503640 10 monclient: init 2024-03-29T11:45:02.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.376+0000 7fbc2f503640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:02.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.376+0000 7fbc2f503640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:02.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.376+0000 7fbc2f503640 10 monclient(hunting): picked mon.noname-a con 0x7fbc280b2800 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:02.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.377+0000 7fbc2f503640 10 monclient(hunting): picked mon.noname-c con 0x7fbc280a9c00 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:02.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.377+0000 7fbc2f503640 10 monclient(hunting): picked mon.noname-b con 0x7fbc280b38c0 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:02.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.377+0000 7fbc2f503640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc28141eb0 con 0x7fbc280b2800 2024-03-29T11:45:02.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.377+0000 7fbc2f503640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc280b22d0 con 0x7fbc280a9c00 2024-03-29T11:45:02.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.377+0000 7fbc2f503640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc28142d30 con 0x7fbc280b38c0 2024-03-29T11:45:02.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.377+0000 7fbc2f503640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:02.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.377+0000 7fbc2f503640 10 monclient(hunting): authenticate will time out at 1115.656815s 2024-03-29T11:45:02.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.379+0000 7fbc2e501640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fbc280b2800 0x7fbc28142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58089/0 (socket says 172.21.15.55:58089) 2024-03-29T11:45:02.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.379+0000 7fbc2e501640 1 -- 172.21.15.55:0/3433021377 learned_addr learned my addr 172.21.15.55:0/3433021377 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265252594 0 0) 0x7fbc28141eb0 con 0x7fbc280b2800 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc14003520 con 0x7fbc280b2800 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578277496 0 0) 0x7fbc280b22d0 con 0x7fbc280a9c00 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc140048a0 con 0x7fbc280a9c00 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3986707863 0 0) 0x7fbc14003520 con 0x7fbc280b2800 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 >> v1:172.21.15.55:6790/0 conn(0x7fbc280b38c0 legacy=0x7fbc2814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 >> v1:172.21.15.171:6789/0 conn(0x7fbc280a9c00 legacy=0x7fbc2813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 1 monclient: found mon.noname-a 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc280aaf90 con 0x7fbc280b2800 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.383+0000 7fbc2d4ff640 10 monclient: _finish_auth 0 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fbc24001670 con 0x7fbc280b2800 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2f503640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2f503640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 10 monclient: dump: 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:02.392 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2f503640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc24001850 con 0x7fbc280b2800 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 1 -- 172.21.15.55:0/3433021377 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fbc24001be0 con 0x7fbc280b2800 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2d4ff640 10 monclient: dump: 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:02.393 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2f503640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.384+0000 7fbc2f503640 4 set_mon_vals no callback set 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.385+0000 7fbc2f503640 10 monclient: shutdown 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.385+0000 7fbc2f503640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.385+0000 7fbc2f503640 1 -- 172.21.15.55:0/3433021377 >> v1:172.21.15.55:6789/0 conn(0x7fbc280b2800 legacy=0x7fbc28142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.385+0000 7fbc2f503640 1 -- 172.21.15.55:0/3433021377 shutdown_connections 2024-03-29T11:45:02.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.385+0000 7fbc2f503640 1 -- 172.21.15.55:0/3433021377 >> 172.21.15.55:0/3433021377 conn(0x7fbc28066a00 msgr2=0x7fbc28066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:02.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.388+0000 7fbc2f503640 1 -- 172.21.15.55:0/3433021377 shutdown_connections 2024-03-29T11:45:02.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.388+0000 7fbc2f503640 1 -- 172.21.15.55:0/3433021377 wait complete. 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.389+0000 7fbc2f503640 10 monclient: build_initial_monmap 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.389+0000 7fbc2f503640 10 monclient: monmap: 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.390+0000 7fbc2f503640 1 librados: starting msgr at 2024-03-29T11:45:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.390+0000 7fbc2f503640 1 librados: starting objecter 2024-03-29T11:45:02.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.390+0000 7fbc2f503640 1 Processor -- start 2024-03-29T11:45:02.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.393+0000 7fbc2f503640 1 -- start start 2024-03-29T11:45:02.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.393+0000 7fbc2f503640 1 librados: setting wanted keys 2024-03-29T11:45:02.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.393+0000 7fbc2f503640 1 librados: calling monclient init 2024-03-29T11:45:02.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.393+0000 7fbc2f503640 10 monclient: init 2024-03-29T11:45:02.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.394+0000 7fbc2f503640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:02.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.394+0000 7fbc2f503640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:45:02.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.395+0000 7fbc2f503640 10 monclient(hunting): picked mon.noname-c con 0x7fbc280b2800 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.397+0000 7fbc2f503640 10 monclient(hunting): picked mon.noname-b con 0x7fbc280b38c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:02.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.397+0000 7fbc2f503640 10 monclient(hunting): picked mon.noname-a con 0x7fbc280ad300 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:02.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.401+0000 7fbc2e501640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fbc280b2800 0x7fbc281da3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:59231/0 (socket says 172.21.15.55:59231) 2024-03-29T11:45:02.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.401+0000 7fbc2e501640 1 -- 172.21.15.55:0/722503263 learned_addr learned my addr 172.21.15.55:0/722503263 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:02.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.401+0000 7fbc2f503640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc280ad790 con 0x7fbc280ad300 2024-03-29T11:45:02.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.401+0000 7fbc2f503640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2813a9b0 con 0x7fbc280b38c0 2024-03-29T11:45:02.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.401+0000 7fbc2f503640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc281daab0 con 0x7fbc280b2800 2024-03-29T11:45:02.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.401+0000 7fbc2f503640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:02.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.401+0000 7fbc2f503640 10 monclient(hunting): authenticate will time out at 1115.681360s 2024-03-29T11:45:02.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 client.?.objecter ms_handle_connect 0x7fbc280ad300 2024-03-29T11:45:02.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:02.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 client.?.objecter ms_handle_connect 0x7fbc280b38c0 2024-03-29T11:45:02.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:02.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 client.?.objecter ms_handle_connect 0x7fbc280b2800 2024-03-29T11:45:02.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:02.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310981685 0 0) 0x7fbc280ad790 con 0x7fbc280ad300 2024-03-29T11:45:02.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.403+0000 7fbc1effd640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c004eb0 con 0x7fbc280ad300 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822908692 0 0) 0x7fbc281daab0 con 0x7fbc280b2800 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc280ad790 con 0x7fbc280b2800 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 231823567 0 0) 0x7fbc0c004eb0 con 0x7fbc280ad300 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc281daab0 con 0x7fbc280ad300 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fbc20001ad0 con 0x7fbc280ad300 2024-03-29T11:45:02.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient(hunting): dump: 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:02.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1823856676 0 0) 0x7fbc281daab0 con 0x7fbc280ad300 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 >> v1:172.21.15.55:6790/0 conn(0x7fbc280b2800 legacy=0x7fbc281da3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 >> v1:172.21.15.171:6789/0 conn(0x7fbc280b38c0 legacy=0x7fbc280b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 monclient: found mon.a 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc280acee0 con 0x7fbc280ad300 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient: _finish_auth 0 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:02.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc20001fe0 con 0x7fbc280ad300 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fbc20001ad0 con 0x7fbc280ad300 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.404+0000 7fbc1effd640 10 monclient: dump: 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:02.417 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.405+0000 7fbc1ffff640 4 set_mon_vals no callback set 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.405+0000 7fbc2f503640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.406+0000 7fbc2f503640 10 monclient: _renew_subs 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.406+0000 7fbc2f503640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.406+0000 7fbc2f503640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc280ad0d0 con 0x7fbc280ad300 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.406+0000 7fbc2f503640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.406+0000 7fbc2f503640 10 monclient: _renew_subs 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.406+0000 7fbc2f503640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.406+0000 7fbc2f503640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2813e980 con 0x7fbc280ad300 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.406+0000 7fbc2f503640 1 librados: init done 2024-03-29T11:45:02.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.408+0000 7fbc1cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:02.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.408+0000 7fbc1cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:02.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.408+0000 7fbc1cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:02.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.408+0000 7fbc1cff9640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc280ad9d0 con 0x7fbc280ad300 2024-03-29T11:45:02.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.411+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fbc20019860 con 0x7fbc280ad300 2024-03-29T11:45:02.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.412+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fbc200018f0 con 0x7fbc280ad300 2024-03-29T11:45:02.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.412+0000 7fbc1effd640 10 client.4329.objecter ms_dispatch 0x7fbc28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:02.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.412+0000 7fbc1effd640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:02.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.412+0000 7fbc1effd640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:02.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.412+0000 7fbc1effd640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-29T11:45:02.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.416+0000 7fbc2e501640 10 monclient: get_auth_request con 0x7fbc0c06e070 auth_method 0 2024-03-29T11:45:02.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.418+0000 7fbc1effd640 10 client.4329.objecter ms_handle_connect 0x7fbc0c06e070 2024-03-29T11:45:02.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.419+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc2004f510 con 0x7fbc280ad300 2024-03-29T11:45:02.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.419+0000 7fbc1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:02.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.419+0000 7fbc1effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:02.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.748+0000 7fbc1cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:02.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.748+0000 7fbc1cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:02.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.748+0000 7fbc1cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:02.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.748+0000 7fbc1cff9640 1 -- 172.21.15.55:0/722503263 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc280ad980 con 0x7fbc280ad300 2024-03-29T11:45:02.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.750+0000 7fbc1effd640 1 -- 172.21.15.55:0/722503263 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fbc2004f510 con 0x7fbc280ad300 2024-03-29T11:45:02.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.750+0000 7fbc1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:02.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.750+0000 7fbc1effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:02.755 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:45:02.755 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.753+0000 7fbc1cff9640 10 librados: watch_flush enter 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.753+0000 7fbc1cff9640 10 librados: watch_flush exit 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.753+0000 7fbc1cff9640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.753+0000 7fbc1cff9640 10 client.4329.objecter successfully canceled tick 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.753+0000 7fbc1cff9640 1 -- 172.21.15.55:0/722503263 >> v1:172.21.15.171:6816/1463197163 conn(0x7fbc0c06e070 legacy=0x7fbc0c070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.753+0000 7fbc1cff9640 10 monclient: shutdown 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.753+0000 7fbc1cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.753+0000 7fbc1cff9640 1 -- 172.21.15.55:0/722503263 >> v1:172.21.15.55:6789/0 conn(0x7fbc280ad300 legacy=0x7fbc281e2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.754+0000 7fbc1cff9640 1 -- 172.21.15.55:0/722503263 shutdown_connections 2024-03-29T11:45:02.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.754+0000 7fbc1cff9640 1 -- 172.21.15.55:0/722503263 >> 172.21.15.55:0/722503263 conn(0x7fbc28066a00 msgr2=0x7fbc28140470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:02.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.754+0000 7fbc1cff9640 1 -- 172.21.15.55:0/722503263 shutdown_connections 2024-03-29T11:45:02.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.754+0000 7fbc1cff9640 1 -- 172.21.15.55:0/722503263 wait complete. 2024-03-29T11:45:02.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:02.754+0000 7fbc1cff9640 1 librados: shutdown 2024-03-29T11:45:02.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:09.584 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:09.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.734+0000 7f9bb2b72640 10 monclient: get_monmap_and_config 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.734+0000 7f9bb2b72640 10 monclient: build_initial_monmap 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.734+0000 7f9bb2b72640 10 monclient: monmap: 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:45:09.735784+0000 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:45:09.735784+0000 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:45:09.739 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:09.740 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.735+0000 7f9bb2b72640 1 Processor -- start 2024-03-29T11:45:09.740 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.735+0000 7f9bb2b72640 1 -- start start 2024-03-29T11:45:09.740 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.735+0000 7f9bb2b72640 10 monclient: init 2024-03-29T11:45:09.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.736+0000 7f9bb2b72640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:09.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.736+0000 7f9bb2b72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:09.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.737+0000 7f9bb2b72640 10 monclient(hunting): picked mon.noname-c con 0x7f9bac1457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:09.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.740+0000 7f9bb2b72640 10 monclient(hunting): picked mon.noname-b con 0x7f9bac13f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:09.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.742+0000 7f9bb2b72640 10 monclient(hunting): picked mon.noname-a con 0x7f9bac0a9d70 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:09.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.742+0000 7f9bb2b72640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac0aaef0 con 0x7f9bac0a9d70 2024-03-29T11:45:09.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.742+0000 7f9bb2b72640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac145bd0 con 0x7f9bac1457f0 2024-03-29T11:45:09.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.742+0000 7f9bb2b72640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac145eb0 con 0x7f9bac13f650 2024-03-29T11:45:09.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.742+0000 7f9bb2b72640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:09.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.742+0000 7f9bb2b72640 10 monclient(hunting): authenticate will time out at 1123.022035s 2024-03-29T11:45:09.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.742+0000 7f9babfff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f9bac13f650 0x7f9bac0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:43735/0 (socket says 172.21.15.55:43735) 2024-03-29T11:45:09.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.742+0000 7f9babfff640 1 -- 172.21.15.55:0/1719556438 learned_addr learned my addr 172.21.15.55:0/1719556438 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:09.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336611964 0 0) 0x7f9bac145eb0 con 0x7f9bac13f650 2024-03-29T11:45:09.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:09.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:09.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:09.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:45:09.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b9c003520 con 0x7f9bac13f650 2024-03-29T11:45:09.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543575111 0 0) 0x7f9bac145bd0 con 0x7f9bac1457f0 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bac145eb0 con 0x7f9bac1457f0 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038881426 0 0) 0x7f9bac0aaef0 con 0x7f9bac0a9d70 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.743+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bac145bd0 con 0x7f9bac0a9d70 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 996514772 0 0) 0x7f9bac145eb0 con 0x7f9bac1457f0 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 >> v1:172.21.15.55:6790/0 conn(0x7f9bac13f650 legacy=0x7f9bac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 >> v1:172.21.15.55:6789/0 conn(0x7f9bac0a9d70 legacy=0x7f9bac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:09.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 1 monclient: found mon.noname-c 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac146340 con 0x7f9bac1457f0 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9ba00015f0 con 0x7f9bac1457f0 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bab7fe640 10 monclient: dump: 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.756 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.757 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:09.757 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:09.757 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:09.757 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:09.757 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:09.757 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:09.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bb2b72640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:45:09.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.747+0000 7f9bb2b72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:09.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.752+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ba0001750 con 0x7f9bac1457f0 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.752+0000 7f9bab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.752+0000 7f9bab7fe640 1 -- 172.21.15.55:0/1719556438 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9ba0001ba0 con 0x7f9bac1457f0 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.752+0000 7f9bab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.752+0000 7f9bab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.752+0000 7f9bab7fe640 10 monclient: dump: 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:09.758 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:09.759 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:09.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.753+0000 7f9bb2b72640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:09.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.753+0000 7f9bb2b72640 4 set_mon_vals no callback set 2024-03-29T11:45:09.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.754+0000 7f9bb2b72640 10 monclient: shutdown 2024-03-29T11:45:09.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.754+0000 7f9bb2b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:09.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.754+0000 7f9bb2b72640 1 -- 172.21.15.55:0/1719556438 >> v1:172.21.15.171:6789/0 conn(0x7f9bac1457f0 legacy=0x7f9bac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:09.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.758+0000 7f9bb2b72640 1 -- 172.21.15.55:0/1719556438 shutdown_connections 2024-03-29T11:45:09.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.758+0000 7f9bb2b72640 1 -- 172.21.15.55:0/1719556438 >> 172.21.15.55:0/1719556438 conn(0x7f9bac066a00 msgr2=0x7f9bac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:09.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.761+0000 7f9bb2b72640 1 -- 172.21.15.55:0/1719556438 shutdown_connections 2024-03-29T11:45:09.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.761+0000 7f9bb2b72640 1 -- 172.21.15.55:0/1719556438 wait complete. 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.762+0000 7f9bb2b72640 10 monclient: build_initial_monmap 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.763+0000 7f9bb2b72640 10 monclient: monmap: 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:45:09.767 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:09.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.763+0000 7f9bb2b72640 1 librados: starting msgr at 2024-03-29T11:45:09.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.763+0000 7f9bb2b72640 1 librados: starting objecter 2024-03-29T11:45:09.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.763+0000 7f9bb2b72640 1 Processor -- start 2024-03-29T11:45:09.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.763+0000 7f9bb2b72640 1 -- start start 2024-03-29T11:45:09.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.764+0000 7f9bb2b72640 1 librados: setting wanted keys 2024-03-29T11:45:09.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.764+0000 7f9bb2b72640 1 librados: calling monclient init 2024-03-29T11:45:09.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.764+0000 7f9bb2b72640 10 monclient: init 2024-03-29T11:45:09.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.765+0000 7f9bb2b72640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:09.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.765+0000 7f9bb2b72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:09.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.768+0000 7f9bb2b72640 10 monclient(hunting): picked mon.noname-b con 0x7f9bac0a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:09.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.769+0000 7f9bb2b72640 10 monclient(hunting): picked mon.noname-c con 0x7f9bac13f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:09.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.769+0000 7f9bb2b72640 10 monclient(hunting): picked mon.noname-a con 0x7f9bac1457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:09.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.769+0000 7f9bb2b72640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac13ff60 con 0x7f9bac1457f0 2024-03-29T11:45:09.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.769+0000 7f9bb2b72640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac141e90 con 0x7f9bac0a9d70 2024-03-29T11:45:09.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.769+0000 7f9bb2b72640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac1d6f60 con 0x7f9bac13f650 2024-03-29T11:45:09.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.769+0000 7f9bb2b72640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:09.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.769+0000 7f9bb2b72640 10 monclient(hunting): authenticate will time out at 1123.049248s 2024-03-29T11:45:09.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.771+0000 7f9bb090e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9bac0a9d70 0x7f9bac142800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:41489/0 (socket says 172.21.15.55:41489) 2024-03-29T11:45:09.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.771+0000 7f9bb090e640 1 -- 172.21.15.55:0/3938274018 learned_addr learned my addr 172.21.15.55:0/3938274018 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:09.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.771+0000 7f9ba97fa640 10 client.?.objecter ms_handle_connect 0x7f9bac0a9d70 2024-03-29T11:45:09.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.771+0000 7f9ba97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:09.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.775+0000 7f9ba97fa640 10 client.?.objecter ms_handle_connect 0x7f9bac1457f0 2024-03-29T11:45:09.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.775+0000 7f9ba97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:09.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.775+0000 7f9ba97fa640 10 client.?.objecter ms_handle_connect 0x7f9bac13f650 2024-03-29T11:45:09.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.775+0000 7f9ba97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:09.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.775+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219046306 0 0) 0x7f9bac13ff60 con 0x7f9bac1457f0 2024-03-29T11:45:09.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.775+0000 7f9ba97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:09.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.775+0000 7f9ba97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b90004eb0 con 0x7f9bac1457f0 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733389846 0 0) 0x7f9bac141e90 con 0x7f9bac0a9d70 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bac13ff60 con 0x7f9bac0a9d70 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179029657 0 0) 0x7f9bac1d6f60 con 0x7f9bac13f650 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:09.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:45:09.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.776+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bac141e90 con 0x7f9bac13f650 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3295235074 0 0) 0x7f9bac141e90 con 0x7f9bac13f650 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bac1d6f60 con 0x7f9bac13f650 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9b980018d0 con 0x7f9bac13f650 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient(hunting): dump: 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.785 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3449894445 0 0) 0x7f9b90004eb0 con 0x7f9bac1457f0 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bac141e90 con 0x7f9bac1457f0 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9ba4001d90 con 0x7f9bac1457f0 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:09.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient(hunting): dump: 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3189502182 0 0) 0x7f9bac1d6f60 con 0x7f9bac13f650 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 >> v1:172.21.15.171:6789/0 conn(0x7f9bac0a9d70 legacy=0x7f9bac142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 >> v1:172.21.15.55:6789/0 conn(0x7f9bac1457f0 legacy=0x7f9bac131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:09.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 monclient: found mon.c 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac142100 con 0x7f9bac13f650 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient: _finish_auth 0 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.778+0000 7f9ba97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9ba97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9bb2b72640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9bb2b72640 10 monclient: _renew_subs 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9bb2b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac132480 con 0x7f9bac13f650 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9bb2b72640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9bb2b72640 10 monclient: _renew_subs 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9bb2b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac13e500 con 0x7f9bac13f650 2024-03-29T11:45:09.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9bb2b72640 1 librados: init done 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b980047e0 con 0x7f9bac13f650 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9ba97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.779+0000 7f9baaffd640 4 set_mon_vals no callback set 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.780+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9b98004af0 con 0x7f9bac13f650 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.780+0000 7f9ba97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.780+0000 7f9ba97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.780+0000 7f9ba97fa640 10 monclient: dump: 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:09.789 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:09.790 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:09.790 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.781+0000 7f9bb2b72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.781+0000 7f9bb2b72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.781+0000 7f9bb2b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.781+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8c005080 con 0x7f9bac13f650 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.785+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f9b9801aac0 con 0x7f9bac13f650 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.786+0000 7f9bb090e640 10 monclient: get_auth_request con 0x7f9b9006df50 auth_method 0 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.786+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f9b98084950 con 0x7f9bac13f650 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.786+0000 7f9ba97fa640 10 client.4352.objecter ms_dispatch 0x7f9bac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.786+0000 7f9ba97fa640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.786+0000 7f9ba97fa640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:09.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.786+0000 7f9ba97fa640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:45:09.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.790+0000 7f9ba97fa640 10 client.4352.objecter ms_handle_connect 0x7f9b9006df50 2024-03-29T11:45:09.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.794+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b98050230 con 0x7f9bac13f650 2024-03-29T11:45:09.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.794+0000 7f9ba97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:09.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:09.794+0000 7f9ba97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:10.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.136+0000 7f9bb2b72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:10.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.136+0000 7f9bb2b72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:10.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.136+0000 7f9bb2b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:10.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.136+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b8c005330 con 0x7f9bac13f650 2024-03-29T11:45:10.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.137+0000 7f9ba97fa640 1 -- 172.21.15.55:0/3938274018 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f9b98052160 con 0x7f9bac13f650 2024-03-29T11:45:10.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.137+0000 7f9ba97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:10.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.137+0000 7f9ba97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:10.143 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:45:10.143 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:10.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.141+0000 7f9bb2b72640 10 librados: watch_flush enter 2024-03-29T11:45:10.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.143+0000 7f9bb2b72640 10 librados: watch_flush exit 2024-03-29T11:45:10.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.143+0000 7f9bb2b72640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:45:10.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.143+0000 7f9bb2b72640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:45:10.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.143+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 >> v1:172.21.15.171:6816/1463197163 conn(0x7f9b9006df50 legacy=0x7f9b90070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:10.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.143+0000 7f9bb2b72640 10 monclient: shutdown 2024-03-29T11:45:10.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.143+0000 7f9bb2b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:10.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.143+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 >> v1:172.21.15.55:6790/0 conn(0x7f9bac13f650 legacy=0x7f9bac1d6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:10.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.143+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 shutdown_connections 2024-03-29T11:45:10.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.143+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 >> 172.21.15.55:0/3938274018 conn(0x7f9bac066a00 msgr2=0x7f9bac135f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:10.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.148+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 shutdown_connections 2024-03-29T11:45:10.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.148+0000 7f9bb2b72640 1 -- 172.21.15.55:0/3938274018 wait complete. 2024-03-29T11:45:10.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:10.149+0000 7f9bb2b72640 1 librados: shutdown 2024-03-29T11:45:10.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:16.978 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:17.131 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.127+0000 7fed114d3640 10 monclient: get_monmap_and_config 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.127+0000 7fed114d3640 10 monclient: build_initial_monmap 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.127+0000 7fed114d3640 10 monclient: monmap: 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:45:17.128373+0000 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:45:17.128373+0000 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:17.132 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:45:17.133 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:45:17.133 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:17.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.130+0000 7fed114d3640 1 Processor -- start 2024-03-29T11:45:17.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.130+0000 7fed114d3640 1 -- start start 2024-03-29T11:45:17.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.130+0000 7fed114d3640 10 monclient: init 2024-03-29T11:45:17.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.132+0000 7fed114d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:17.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.132+0000 7fed114d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:17.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.132+0000 7fed114d3640 10 monclient(hunting): picked mon.noname-c con 0x7fed0c1457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:17.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.134+0000 7fed114d3640 10 monclient(hunting): picked mon.noname-a con 0x7fed0c13f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:17.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.135+0000 7fed0affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fed0c1457f0 0x7fed0c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:41543/0 (socket says 172.21.15.55:41543) 2024-03-29T11:45:17.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.135+0000 7fed114d3640 10 monclient(hunting): picked mon.noname-b con 0x7fed0c0a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:17.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.135+0000 7fed114d3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c0aaef0 con 0x7fed0c13f650 2024-03-29T11:45:17.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.135+0000 7fed114d3640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c145bd0 con 0x7fed0c1457f0 2024-03-29T11:45:17.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.135+0000 7fed114d3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c145eb0 con 0x7fed0c0a9d70 2024-03-29T11:45:17.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.135+0000 7fed114d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:17.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.135+0000 7fed114d3640 10 monclient(hunting): authenticate will time out at 1130.414878s 2024-03-29T11:45:17.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.135+0000 7fed0affd640 1 -- 172.21.15.55:0/1589301289 learned_addr learned my addr 172.21.15.55:0/1589301289 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:17.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.137+0000 7fed09ffb640 1 -- 172.21.15.55:0/1589301289 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241975639 0 0) 0x7fed0c145bd0 con 0x7fed0c1457f0 2024-03-29T11:45:17.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.137+0000 7fed09ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:17.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.137+0000 7fed09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:17.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.137+0000 7fed09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:17.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.137+0000 7fed09ffb640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:45:17.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.137+0000 7fed09ffb640 1 -- 172.21.15.55:0/1589301289 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf0003520 con 0x7fed0c1457f0 2024-03-29T11:45:17.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.138+0000 7fed09ffb640 1 -- 172.21.15.55:0/1589301289 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2403355501 0 0) 0x7fecf0003520 con 0x7fed0c1457f0 2024-03-29T11:45:17.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.138+0000 7fed09ffb640 1 -- 172.21.15.55:0/1589301289 >> v1:172.21.15.55:6790/0 conn(0x7fed0c0a9d70 legacy=0x7fed0c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:17.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.138+0000 7fed09ffb640 1 -- 172.21.15.55:0/1589301289 >> v1:172.21.15.55:6789/0 conn(0x7fed0c13f650 legacy=0x7fed0c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.138+0000 7fed09ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:17.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 1 monclient: found mon.noname-c 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 1 -- 172.21.15.55:0/1589301289 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c146340 con 0x7fed0c1457f0 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 1 -- 172.21.15.55:0/1589301289 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fecfc001690 con 0x7fed0c1457f0 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: dump: 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.147 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed114d3640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed114d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 1 -- 172.21.15.55:0/1589301289 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecfc001860 con 0x7fed0c1457f0 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:17.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 1 -- 172.21.15.55:0/1589301289 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fecfc001c70 con 0x7fed0c1457f0 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed09ffb640 10 monclient: dump: 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:17.149 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:17.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed114d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:17.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.139+0000 7fed114d3640 4 set_mon_vals no callback set 2024-03-29T11:45:17.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.140+0000 7fed114d3640 10 monclient: shutdown 2024-03-29T11:45:17.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.140+0000 7fed114d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:17.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.140+0000 7fed114d3640 1 -- 172.21.15.55:0/1589301289 >> v1:172.21.15.171:6789/0 conn(0x7fed0c1457f0 legacy=0x7fed0c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.140+0000 7fed114d3640 1 -- 172.21.15.55:0/1589301289 shutdown_connections 2024-03-29T11:45:17.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.140+0000 7fed114d3640 1 -- 172.21.15.55:0/1589301289 >> 172.21.15.55:0/1589301289 conn(0x7fed0c066a00 msgr2=0x7fed0c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:17.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.140+0000 7fed114d3640 1 -- 172.21.15.55:0/1589301289 shutdown_connections 2024-03-29T11:45:17.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.140+0000 7fed114d3640 1 -- 172.21.15.55:0/1589301289 wait complete. 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.141+0000 7fed114d3640 10 monclient: build_initial_monmap 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.141+0000 7fed114d3640 10 monclient: monmap: 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:45:17.151 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:45:17.152 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:17.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.142+0000 7fed114d3640 1 librados: starting msgr at 2024-03-29T11:45:17.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.142+0000 7fed114d3640 1 librados: starting objecter 2024-03-29T11:45:17.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.142+0000 7fed114d3640 1 Processor -- start 2024-03-29T11:45:17.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.142+0000 7fed114d3640 1 -- start start 2024-03-29T11:45:17.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.142+0000 7fed114d3640 1 librados: setting wanted keys 2024-03-29T11:45:17.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.142+0000 7fed114d3640 1 librados: calling monclient init 2024-03-29T11:45:17.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.142+0000 7fed114d3640 10 monclient: init 2024-03-29T11:45:17.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.143+0000 7fed114d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:17.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.143+0000 7fed114d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:17.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.143+0000 7fed114d3640 10 monclient(hunting): picked mon.noname-a con 0x7fed0c0a9d70 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:17.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.143+0000 7fed114d3640 10 monclient(hunting): picked mon.noname-b con 0x7fed0c13f650 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:17.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.145+0000 7fed114d3640 10 monclient(hunting): picked mon.noname-c con 0x7fed0c1457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:17.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.145+0000 7fed114d3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c06e810 con 0x7fed0c0a9d70 2024-03-29T11:45:17.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.145+0000 7fed114d3640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c06fb50 con 0x7fed0c13f650 2024-03-29T11:45:17.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.146+0000 7fed114d3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c06ed30 con 0x7fed0c1457f0 2024-03-29T11:45:17.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.146+0000 7fed114d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:17.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.146+0000 7fed114d3640 10 monclient(hunting): authenticate will time out at 1130.425694s 2024-03-29T11:45:17.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.151+0000 7fed0a7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fed0c13f650 0x7fed0c14e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:41615/0 (socket says 172.21.15.55:41615) 2024-03-29T11:45:17.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.151+0000 7fed0a7fc640 1 -- 172.21.15.55:0/2064487931 learned_addr learned my addr 172.21.15.55:0/2064487931 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:17.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 10 client.?.objecter ms_handle_connect 0x7fed0c13f650 2024-03-29T11:45:17.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:17.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892417103 0 0) 0x7fed0c06fb50 con 0x7fed0c13f650 2024-03-29T11:45:17.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:17.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:17.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:17.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:45:17.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf4004eb0 con 0x7fed0c13f650 2024-03-29T11:45:17.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 10 client.?.objecter ms_handle_connect 0x7fed0c1457f0 2024-03-29T11:45:17.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.152+0000 7fed037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:17.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044918149 0 0) 0x7fed0c06ed30 con 0x7fed0c1457f0 2024-03-29T11:45:17.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:17.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:17.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:17.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:45:17.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed0c06fb50 con 0x7fed0c1457f0 2024-03-29T11:45:17.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 client.?.objecter ms_handle_connect 0x7fed0c0a9d70 2024-03-29T11:45:17.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3654520158 0 0) 0x7fed0c06fb50 con 0x7fed0c1457f0 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed0c06ed30 con 0x7fed0c1457f0 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fed04001890 con 0x7fed0c1457f0 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 monclient(hunting): dump: 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:17.160 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2226552690 0 0) 0x7fecf4004eb0 con 0x7fed0c13f650 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.153+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed0c06fb50 con 0x7fed0c13f650 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fecf8001a30 con 0x7fed0c13f650 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 10 monclient(hunting): dump: 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.161 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1306957242 0 0) 0x7fed0c06ed30 con 0x7fed0c1457f0 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 >> v1:172.21.15.171:6789/0 conn(0x7fed0c13f650 legacy=0x7fed0c14e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 >> v1:172.21.15.55:6789/0 conn(0x7fed0c0a9d70 legacy=0x7fed0c06e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 1 monclient: found mon.c 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c06fdc0 con 0x7fed0c1457f0 2024-03-29T11:45:17.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed114d3640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed114d3640 10 monclient: _renew_subs 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed114d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed114d3640 1 -- 172.21.15.55:0/2064487931 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c06e9c0 con 0x7fed0c1457f0 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed114d3640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed114d3640 10 monclient: _renew_subs 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed114d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed114d3640 1 -- 172.21.15.55:0/2064487931 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c159bc0 con 0x7fed0c1457f0 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.154+0000 7fed114d3640 1 librados: init done 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.156+0000 7fed017fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.156+0000 7fed017fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.156+0000 7fed017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:17.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.156+0000 7fed017fa640 1 -- 172.21.15.55:0/2064487931 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece8005080 con 0x7fed0c1457f0 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.157+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed040019f0 con 0x7fed0c1457f0 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.157+0000 7fed037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.157+0000 7fed097fa640 4 set_mon_vals no callback set 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.157+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fed04002000 con 0x7fed0c1457f0 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.157+0000 7fed037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.157+0000 7fed037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.157+0000 7fed037fe640 10 monclient: dump: 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:17.165 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.157+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fed04019a80 con 0x7fed0c1457f0 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.158+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fed040839f0 con 0x7fed0c1457f0 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.158+0000 7fed037fe640 10 client.4355.objecter ms_dispatch 0x7fed0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.158+0000 7fed037fe640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.158+0000 7fed037fe640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.158+0000 7fed037fe640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-29T11:45:17.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.158+0000 7fed0affd640 10 monclient: get_auth_request con 0x7fecf406df30 auth_method 0 2024-03-29T11:45:17.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.162+0000 7fed037fe640 10 client.4355.objecter ms_handle_connect 0x7fecf406df30 2024-03-29T11:45:17.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.170+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed0404f240 con 0x7fed0c1457f0 2024-03-29T11:45:17.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.170+0000 7fed037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:17.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.170+0000 7fed037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:17.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.494+0000 7fed017fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:17.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.494+0000 7fed017fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:17.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.495+0000 7fed017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:17.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.495+0000 7fed017fa640 1 -- 172.21.15.55:0/2064487931 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fece8005330 con 0x7fed0c1457f0 2024-03-29T11:45:17.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.498+0000 7fed037fe640 1 -- 172.21.15.55:0/2064487931 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fed04051340 con 0x7fed0c1457f0 2024-03-29T11:45:17.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.498+0000 7fed037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:17.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.498+0000 7fed037fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:17.503 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:45:17.504 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:17.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 10 librados: watch_flush enter 2024-03-29T11:45:17.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 10 librados: watch_flush exit 2024-03-29T11:45:17.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-29T11:45:17.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 10 client.4355.objecter successfully canceled tick 2024-03-29T11:45:17.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 1 -- 172.21.15.55:0/2064487931 >> v1:172.21.15.171:6816/1463197163 conn(0x7fecf406df30 legacy=0x7fecf40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 10 monclient: shutdown 2024-03-29T11:45:17.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:17.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 1 -- 172.21.15.55:0/2064487931 >> v1:172.21.15.55:6790/0 conn(0x7fed0c1457f0 legacy=0x7fed0c14ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 1 -- 172.21.15.55:0/2064487931 shutdown_connections 2024-03-29T11:45:17.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.502+0000 7fed017fa640 1 -- 172.21.15.55:0/2064487931 >> 172.21.15.55:0/2064487931 conn(0x7fed0c066a00 msgr2=0x7fed0c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.506+0000 7fed017fa640 1 -- 172.21.15.55:0/2064487931 shutdown_connections 2024-03-29T11:45:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.506+0000 7fed017fa640 1 -- 172.21.15.55:0/2064487931 wait complete. 2024-03-29T11:45:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:17.507+0000 7fed017fa640 1 librados: shutdown 2024-03-29T11:45:17.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:24.335 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:24.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.480+0000 7f6ef1d8e640 10 monclient: get_monmap_and_config 2024-03-29T11:45:24.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.480+0000 7f6ef1d8e640 10 monclient: build_initial_monmap 2024-03-29T11:45:24.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.480+0000 7f6ef1d8e640 10 monclient: monmap: 2024-03-29T11:45:24.485 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:24.485 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:24.485 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:45:24.481788+0000 2024-03-29T11:45:24.485 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:45:24.481788+0000 2024-03-29T11:45:24.485 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:24.486 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:24.486 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:24.486 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:45:24.486 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:45:24.486 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.481+0000 7f6ef1d8e640 1 Processor -- start 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.481+0000 7f6ef1d8e640 1 -- start start 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.481+0000 7f6ef1d8e640 10 monclient: init 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 10 monclient(hunting): picked mon.noname-b con 0x7f6eec0b4790 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 10 monclient(hunting): picked mon.noname-a con 0x7f6eec0aea90 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 10 monclient(hunting): picked mon.noname-c con 0x7f6eec0b39f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec0b0c60 con 0x7f6eec0aea90 2024-03-29T11:45:24.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec0af020 con 0x7f6eec0b39f0 2024-03-29T11:45:24.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec13fbd0 con 0x7f6eec0b4790 2024-03-29T11:45:24.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:24.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6ef1d8e640 10 monclient(hunting): authenticate will time out at 1137.762342s 2024-03-29T11:45:24.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.482+0000 7f6eeb7fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f6eec0b4790 0x7f6eec0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:46172/0 (socket says 172.21.15.55:46172) 2024-03-29T11:45:24.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.483+0000 7f6eeb7fe640 1 -- 172.21.15.55:0/3706932969 learned_addr learned my addr 172.21.15.55:0/3706932969 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.483+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677418006 0 0) 0x7f6eec0af020 con 0x7f6eec0b39f0 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.483+0000 7f6eea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.483+0000 7f6eea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.483+0000 7f6eea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.483+0000 7f6eea7fc640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.483+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed4003520 con 0x7f6eec0b39f0 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439220134 0 0) 0x7f6eec13fbd0 con 0x7f6eec0b4790 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec0af020 con 0x7f6eec0b4790 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3401789851 0 0) 0x7f6ed4003520 con 0x7f6eec0b39f0 2024-03-29T11:45:24.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 >> v1:172.21.15.55:6790/0 conn(0x7f6eec0b4790 legacy=0x7f6eec0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 >> v1:172.21.15.55:6789/0 conn(0x7f6eec0aea90 legacy=0x7f6eec0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 1 monclient: found mon.noname-c 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec13fd60 con 0x7f6eec0b39f0 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6ee4001990 con 0x7f6eec0b39f0 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6eea7fc640 10 monclient: dump: 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:24.491 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6ef1d8e640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.484+0000 7f6ef1d8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.485+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ee4001af0 con 0x7f6eec0b39f0 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.485+0000 7f6eea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.485+0000 7f6eea7fc640 1 -- 172.21.15.55:0/3706932969 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6ee4001f40 con 0x7f6eec0b39f0 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.485+0000 7f6eea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:24.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.485+0000 7f6eea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.485+0000 7f6eea7fc640 10 monclient: dump: 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.485+0000 7f6ef1d8e640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:24.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.485+0000 7f6ef1d8e640 4 set_mon_vals no callback set 2024-03-29T11:45:24.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.486+0000 7f6ef1d8e640 10 monclient: shutdown 2024-03-29T11:45:24.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.486+0000 7f6ef1d8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:24.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.486+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/3706932969 >> v1:172.21.15.171:6789/0 conn(0x7f6eec0b39f0 legacy=0x7f6eec13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:24.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.486+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/3706932969 shutdown_connections 2024-03-29T11:45:24.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.486+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/3706932969 >> 172.21.15.55:0/3706932969 conn(0x7f6eec066a00 msgr2=0x7f6eec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.486+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/3706932969 shutdown_connections 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.486+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/3706932969 wait complete. 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.487+0000 7f6ef1d8e640 10 monclient: build_initial_monmap 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.487+0000 7f6ef1d8e640 10 monclient: monmap: 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:24.504 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.487+0000 7f6ef1d8e640 1 librados: starting msgr at 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.487+0000 7f6ef1d8e640 1 librados: starting objecter 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.487+0000 7f6ef1d8e640 1 Processor -- start 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 1 -- start start 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 1 librados: setting wanted keys 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 1 librados: calling monclient init 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 10 monclient: init 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:24.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 10 monclient(hunting): picked mon.noname-b con 0x7f6eec0b39f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 10 monclient(hunting): picked mon.noname-c con 0x7f6eec0b4790 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 10 monclient(hunting): picked mon.noname-a con 0x7f6eec137270 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec1380f0 con 0x7f6eec137270 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec138320 con 0x7f6eec0b39f0 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec136640 con 0x7f6eec0b4790 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6ef1d8e640 10 monclient(hunting): authenticate will time out at 1137.768400s 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6eebfff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f6eec137270 0x7f6eec1e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:39860/0 (socket says 172.21.15.55:39860) 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.488+0000 7f6eebfff640 1 -- 172.21.15.55:0/2329971599 learned_addr learned my addr 172.21.15.55:0/2329971599 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 client.?.objecter ms_handle_connect 0x7f6eec0b39f0 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:24.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 client.?.objecter ms_handle_connect 0x7f6eec0b4790 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 client.?.objecter ms_handle_connect 0x7f6eec137270 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434437910 0 0) 0x7f6eec136640 con 0x7f6eec0b4790 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed8004eb0 con 0x7f6eec0b4790 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855182419 0 0) 0x7f6eec138320 con 0x7f6eec0b39f0 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:24.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.489+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec136640 con 0x7f6eec0b39f0 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154447308 0 0) 0x7f6ed8004eb0 con 0x7f6eec0b4790 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eec138320 con 0x7f6eec0b4790 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2250056750 0 0) 0x7f6eec136640 con 0x7f6eec0b39f0 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ed8004eb0 con 0x7f6eec0b39f0 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6edc001930 con 0x7f6eec0b4790 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): dump: 2024-03-29T11:45:24.508 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6ee00017b0 con 0x7f6eec0b39f0 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:24.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): dump: 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:24.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800435135 0 0) 0x7f6eec1380f0 con 0x7f6eec137270 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.490+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec136640 con 0x7f6eec137270 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3574838329 0 0) 0x7f6eec138320 con 0x7f6eec0b4790 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 >> v1:172.21.15.171:6789/0 conn(0x7f6eec0b39f0 legacy=0x7f6eec136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 >> v1:172.21.15.55:6789/0 conn(0x7f6eec137270 legacy=0x7f6eec1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 1 monclient: found mon.c 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:24.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec1410c0 con 0x7f6eec0b4790 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ed3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ef1d8e640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ef1d8e640 10 monclient: _renew_subs 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ef1d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec1d4750 con 0x7f6eec0b4790 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ef1d8e640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ef1d8e640 10 monclient: _renew_subs 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ef1d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec1d4c80 con 0x7f6eec0b4790 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.491+0000 7f6ef1d8e640 1 librados: init done 2024-03-29T11:45:24.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.492+0000 7f6ef1d8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.492+0000 7f6ef1d8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.493+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6edc001a90 con 0x7f6eec0b4790 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.493+0000 7f6ef1d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.493+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec075f10 con 0x7f6eec0b4790 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.493+0000 7f6ed3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.493+0000 7f6ee97fa640 4 set_mon_vals no callback set 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.493+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6edc003810 con 0x7f6eec0b4790 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.493+0000 7f6ed3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.493+0000 7f6ed3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.493+0000 7f6ed3fff640 10 monclient: dump: 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:24.513 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.496+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f6edc003970 con 0x7f6eec0b4790 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.497+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f6edc083840 con 0x7f6eec0b4790 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.497+0000 7f6eeb7fe640 10 monclient: get_auth_request con 0x7f6ed806e070 auth_method 0 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.497+0000 7f6ed3fff640 10 client.4361.objecter ms_dispatch 0x7f6eec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:24.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.497+0000 7f6ed3fff640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:24.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.497+0000 7f6ed3fff640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:24.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.497+0000 7f6ed3fff640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T11:45:24.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.498+0000 7f6ed3fff640 10 client.4361.objecter ms_handle_connect 0x7f6ed806e070 2024-03-29T11:45:24.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.504+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6edc04f180 con 0x7f6eec0b4790 2024-03-29T11:45:24.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.504+0000 7f6ed3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:24.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.504+0000 7f6ed3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:24.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.876+0000 7f6ef1d8e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:24.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.876+0000 7f6ef1d8e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:24.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.876+0000 7f6ef1d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:24.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.876+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6eec074eb0 con 0x7f6eec0b4790 2024-03-29T11:45:24.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.878+0000 7f6ed3fff640 1 -- 172.21.15.55:0/2329971599 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f6edc0510b0 con 0x7f6eec0b4790 2024-03-29T11:45:24.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.878+0000 7f6ed3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:24.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.878+0000 7f6ed3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:24.883 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:45:24.883 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:24.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.883+0000 7f6ef1d8e640 10 librados: watch_flush enter 2024-03-29T11:45:24.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.883+0000 7f6ef1d8e640 10 librados: watch_flush exit 2024-03-29T11:45:24.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.883+0000 7f6ef1d8e640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T11:45:24.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.883+0000 7f6ef1d8e640 10 client.4361.objecter successfully canceled tick 2024-03-29T11:45:24.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.883+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 >> v1:172.21.15.171:6816/1463197163 conn(0x7f6ed806e070 legacy=0x7f6ed8070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:24.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.884+0000 7f6ef1d8e640 10 monclient: shutdown 2024-03-29T11:45:24.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.884+0000 7f6ef1d8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:24.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.884+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 >> v1:172.21.15.55:6790/0 conn(0x7f6eec0b4790 legacy=0x7f6eec1d4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:24.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.884+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 shutdown_connections 2024-03-29T11:45:24.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.884+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 >> 172.21.15.55:0/2329971599 conn(0x7f6eec066a00 msgr2=0x7f6eec13f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:24.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.884+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 shutdown_connections 2024-03-29T11:45:24.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.884+0000 7f6ef1d8e640 1 -- 172.21.15.55:0/2329971599 wait complete. 2024-03-29T11:45:24.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:24.884+0000 7f6ef1d8e640 1 librados: shutdown 2024-03-29T11:45:24.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:31.715 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:31.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.861+0000 7fd8281e8640 10 monclient: get_monmap_and_config 2024-03-29T11:45:31.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.861+0000 7fd8281e8640 10 monclient: build_initial_monmap 2024-03-29T11:45:31.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.861+0000 7fd8281e8640 10 monclient: monmap: 2024-03-29T11:45:31.866 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:31.866 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:31.866 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:45:31.862446+0000 2024-03-29T11:45:31.866 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:45:31.862446+0000 2024-03-29T11:45:31.867 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:31.867 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:31.867 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:31.867 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:45:31.867 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:45:31.867 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:31.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.862+0000 7fd8281e8640 1 Processor -- start 2024-03-29T11:45:31.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.862+0000 7fd8281e8640 1 -- start start 2024-03-29T11:45:31.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.862+0000 7fd8281e8640 10 monclient: init 2024-03-29T11:45:31.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:31.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:45:31.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 10 monclient(hunting): picked mon.noname-b con 0x7fd8200b0770 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:31.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 10 monclient(hunting): picked mon.noname-c con 0x7fd82014a700 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:31.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 10 monclient(hunting): picked mon.noname-a con 0x7fd8200ac5a0 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:31.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200aba70 con 0x7fd8200ac5a0 2024-03-29T11:45:31.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200aecb0 con 0x7fd82014a700 2024-03-29T11:45:31.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200af150 con 0x7fd8200b0770 2024-03-29T11:45:31.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:31.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd8281e8640 10 monclient(hunting): authenticate will time out at 1145.143247s 2024-03-29T11:45:31.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd826785640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fd8200ac5a0 0x7fd820151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47674/0 (socket says 172.21.15.55:47674) 2024-03-29T11:45:31.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd826785640 1 -- 172.21.15.55:0/2143402988 learned_addr learned my addr 172.21.15.55:0/2143402988 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:31.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.863+0000 7fd825f84640 1 --1- 172.21.15.55:0/2143402988 >> v1:172.21.15.55:6790/0 conn(0x7fd8200b0770 0x7fd820149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:48968/0 (socket says 172.21.15.55:48968) 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329406539 0 0) 0x7fd8200aba70 con 0x7fd8200ac5a0 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 10 monclient(hunting): _negotiate 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd80c003520 con 0x7fd8200ac5a0 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106817635 0 0) 0x7fd8200af150 con 0x7fd8200b0770 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 10 monclient(hunting): _negotiate 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.864+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8200aba70 con 0x7fd8200b0770 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3623973135 0 0) 0x7fd80c003520 con 0x7fd8200ac5a0 2024-03-29T11:45:31.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 >> v1:172.21.15.55:6790/0 conn(0x7fd8200b0770 legacy=0x7fd820149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 >> v1:172.21.15.171:6789/0 conn(0x7fd82014a700 legacy=0x7fd8200abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: _finish_hunting 0 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 1 monclient: found mon.noname-a 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8200ac9d0 con 0x7fd8200ac5a0 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: _finish_auth 0 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: _check_auth_tickets 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd814001690 con 0x7fd8200ac5a0 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd8281e8640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd8281e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:31.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: dump: 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd8281e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:31.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8140017f0 con 0x7fd8200ac5a0 2024-03-29T11:45:31.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:31.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 1 -- 172.21.15.55:0/2143402988 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd814001c00 con 0x7fd8200ac5a0 2024-03-29T11:45:31.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:31.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:31.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd824f82640 10 monclient: dump: 2024-03-29T11:45:31.885 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:31.885 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd8281e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.865+0000 7fd8281e8640 4 set_mon_vals no callback set 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.866+0000 7fd8281e8640 10 monclient: shutdown 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.866+0000 7fd8281e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.866+0000 7fd8281e8640 1 -- 172.21.15.55:0/2143402988 >> v1:172.21.15.55:6789/0 conn(0x7fd8200ac5a0 legacy=0x7fd820151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:31.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.866+0000 7fd8281e8640 1 -- 172.21.15.55:0/2143402988 shutdown_connections 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.866+0000 7fd8281e8640 1 -- 172.21.15.55:0/2143402988 >> 172.21.15.55:0/2143402988 conn(0x7fd82006b660 msgr2=0x7fd82006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.866+0000 7fd8281e8640 1 -- 172.21.15.55:0/2143402988 shutdown_connections 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.867+0000 7fd8281e8640 1 -- 172.21.15.55:0/2143402988 wait complete. 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.867+0000 7fd8281e8640 10 monclient: build_initial_monmap 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.867+0000 7fd8281e8640 10 monclient: monmap: 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:45:31.887 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.868+0000 7fd8281e8640 1 librados: starting msgr at 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.868+0000 7fd8281e8640 1 librados: starting objecter 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.868+0000 7fd8281e8640 1 Processor -- start 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.868+0000 7fd8281e8640 1 -- start start 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.868+0000 7fd8281e8640 1 librados: setting wanted keys 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.868+0000 7fd8281e8640 1 librados: calling monclient init 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.868+0000 7fd8281e8640 10 monclient: init 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 10 monclient(hunting): picked mon.noname-c con 0x7fd8200ac5a0 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 10 monclient(hunting): picked mon.noname-b con 0x7fd8200b0770 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 10 monclient(hunting): picked mon.noname-a con 0x7fd82014a700 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd820128510 con 0x7fd82014a700 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8201459a0 con 0x7fd8200b0770 2024-03-29T11:45:31.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd820145c50 con 0x7fd8200ac5a0 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.869+0000 7fd8281e8640 10 monclient(hunting): authenticate will time out at 1145.149332s 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd826785640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fd82014a700 0x7fd820145290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47834/0 (socket says 172.21.15.55:47834) 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd826785640 1 -- 172.21.15.55:0/160462065 learned_addr learned my addr 172.21.15.55:0/160462065 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd8127fc640 10 client.?.objecter ms_handle_connect 0x7fd82014a700 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd8127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd8127fc640 10 client.?.objecter ms_handle_connect 0x7fd8200b0770 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd8127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098638666 0 0) 0x7fd820128510 con 0x7fd82014a700 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd8127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd8127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.870+0000 7fd8127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:45:31.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd804004eb0 con 0x7fd82014a700 2024-03-29T11:45:31.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550439384 0 0) 0x7fd8201459a0 con 0x7fd8200b0770 2024-03-29T11:45:31.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:31.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:31.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:31.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:45:31.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd820128510 con 0x7fd8200b0770 2024-03-29T11:45:31.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1382483019 0 0) 0x7fd804004eb0 con 0x7fd82014a700 2024-03-29T11:45:31.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8201459a0 con 0x7fd82014a700 2024-03-29T11:45:31.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd814001bc0 con 0x7fd82014a700 2024-03-29T11:45:31.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): dump: 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2008489849 0 0) 0x7fd820128510 con 0x7fd8200b0770 2024-03-29T11:45:31.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd804004eb0 con 0x7fd8200b0770 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 client.?.objecter ms_handle_connect 0x7fd8200ac5a0 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd81c001790 con 0x7fd8200b0770 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient(hunting): dump: 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:31.893 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2929819026 0 0) 0x7fd8201459a0 con 0x7fd82014a700 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 >> v1:172.21.15.55:6790/0 conn(0x7fd8200ac5a0 legacy=0x7fd8201267b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 >> v1:172.21.15.171:6789/0 conn(0x7fd8200b0770 legacy=0x7fd820144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 monclient: found mon.a 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd820126290 con 0x7fd82014a700 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8281e8640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:45:31.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8281e8640 10 monclient: _renew_subs 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8281e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.871+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd820125e40 con 0x7fd82014a700 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8281e8640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8281e8640 10 monclient: _renew_subs 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8281e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8201dc350 con 0x7fd82014a700 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8281e8640 1 librados: init done 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd814004a90 con 0x7fd82014a700 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd813fff640 4 set_mon_vals no callback set 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd814004e20 con 0x7fd82014a700 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.872+0000 7fd8127fc640 10 monclient: dump: 2024-03-29T11:45:31.895 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.874+0000 7fd8281e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.874+0000 7fd8281e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.874+0000 7fd8281e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.874+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd808005080 con 0x7fd82014a700 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.875+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fd814004f80 con 0x7fd82014a700 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.876+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fd814084af0 con 0x7fd82014a700 2024-03-29T11:45:31.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.876+0000 7fd8127fc640 10 client.4347.objecter ms_dispatch 0x7fd820000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:31.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.876+0000 7fd8127fc640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:31.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.876+0000 7fd8127fc640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:31.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.876+0000 7fd8127fc640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T11:45:31.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.876+0000 7fd825f84640 10 monclient: get_auth_request con 0x7fd80406de90 auth_method 0 2024-03-29T11:45:31.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.878+0000 7fd8127fc640 10 client.4347.objecter ms_handle_connect 0x7fd80406de90 2024-03-29T11:45:31.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.885+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8140503d0 con 0x7fd82014a700 2024-03-29T11:45:31.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.885+0000 7fd8127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:31.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:31.885+0000 7fd8127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:32.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.293+0000 7fd8281e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:32.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.293+0000 7fd8281e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:32.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.293+0000 7fd8281e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:32.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.293+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd808005330 con 0x7fd82014a700 2024-03-29T11:45:32.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.295+0000 7fd8127fc640 1 -- 172.21.15.55:0/160462065 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fd814052300 con 0x7fd82014a700 2024-03-29T11:45:32.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.295+0000 7fd8127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:32.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.295+0000 7fd8127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:32.300 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:45:32.300 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:32.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 10 librados: watch_flush enter 2024-03-29T11:45:32.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 10 librados: watch_flush exit 2024-03-29T11:45:32.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T11:45:32.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 10 client.4347.objecter successfully canceled tick 2024-03-29T11:45:32.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 >> v1:172.21.15.171:6816/1463197163 conn(0x7fd80406de90 legacy=0x7fd804070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:32.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 10 monclient: shutdown 2024-03-29T11:45:32.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:32.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 >> v1:172.21.15.55:6789/0 conn(0x7fd82014a700 legacy=0x7fd820145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:32.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 shutdown_connections 2024-03-29T11:45:32.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 >> 172.21.15.55:0/160462065 conn(0x7fd82006b660 msgr2=0x7fd820071640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:32.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 shutdown_connections 2024-03-29T11:45:32.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.299+0000 7fd8281e8640 1 -- 172.21.15.55:0/160462065 wait complete. 2024-03-29T11:45:32.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:32.300+0000 7fd8281e8640 1 librados: shutdown 2024-03-29T11:45:32.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:39.127 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:39.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.277+0000 7f3fa9932640 10 monclient: get_monmap_and_config 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.277+0000 7f3fa9932640 10 monclient: build_initial_monmap 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.277+0000 7f3fa9932640 10 monclient: monmap: 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:45:39.278501+0000 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:45:39.278501+0000 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:39.282 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:45:39.283 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:45:39.283 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:39.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.278+0000 7f3fa9932640 1 Processor -- start 2024-03-29T11:45:39.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.278+0000 7f3fa9932640 1 -- start start 2024-03-29T11:45:39.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.278+0000 7f3fa9932640 10 monclient: init 2024-03-29T11:45:39.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:39.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:39.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa406f050 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:39.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa4070240 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:39.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa406f430 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa413f400 con 0x7f3fa406f050 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa41429c0 con 0x7f3fa406f430 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa406ff80 con 0x7f3fa4070240 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa9932640 10 monclient(hunting): authenticate will time out at 1152.559253s 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa27fc640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f3fa4070240 0x7f3fa40ae860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:59638/0 (socket says 172.21.15.55:59638) 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.279+0000 7f3fa27fc640 1 -- 172.21.15.55:0/383219741 learned_addr learned my addr 172.21.15.55:0/383219741 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982934618 0 0) 0x7f3fa406ff80 con 0x7f3fa4070240 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8c003520 con 0x7f3fa4070240 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125339027 0 0) 0x7f3fa41429c0 con 0x7f3fa406f430 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8c003840 con 0x7f3fa406f430 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.280+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1116055276 0 0) 0x7f3f8c003520 con 0x7f3fa4070240 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 >> v1:172.21.15.171:6789/0 conn(0x7f3fa406f430 legacy=0x7f3fa414a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 >> v1:172.21.15.55:6789/0 conn(0x7f3fa406f050 legacy=0x7f3fa41422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:45:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa40aef70 con 0x7f3fa4070240 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa9932640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa9932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3f94001670 con 0x7f3fa4070240 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: dump: 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.289 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa9932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f94001850 con 0x7f3fa4070240 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 1 -- 172.21.15.55:0/383219741 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3f94001670 con 0x7f3fa4070240 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa1ffb640 10 monclient: dump: 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:39.290 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa9932640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:39.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.281+0000 7f3fa9932640 4 set_mon_vals no callback set 2024-03-29T11:45:39.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.282+0000 7f3fa9932640 10 monclient: shutdown 2024-03-29T11:45:39.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.282+0000 7f3fa9932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:39.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.282+0000 7f3fa9932640 1 -- 172.21.15.55:0/383219741 >> v1:172.21.15.55:6790/0 conn(0x7f3fa4070240 legacy=0x7f3fa40ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.282+0000 7f3fa9932640 1 -- 172.21.15.55:0/383219741 shutdown_connections 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.282+0000 7f3fa9932640 1 -- 172.21.15.55:0/383219741 >> 172.21.15.55:0/383219741 conn(0x7f3fa40b4930 msgr2=0x7f3fa40b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.282+0000 7f3fa9932640 1 -- 172.21.15.55:0/383219741 shutdown_connections 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.282+0000 7f3fa9932640 1 -- 172.21.15.55:0/383219741 wait complete. 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.283+0000 7f3fa9932640 10 monclient: build_initial_monmap 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.283+0000 7f3fa9932640 10 monclient: monmap: 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:39.296 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.284+0000 7f3fa9932640 1 librados: starting msgr at 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.284+0000 7f3fa9932640 1 librados: starting objecter 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.284+0000 7f3fa9932640 1 Processor -- start 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.284+0000 7f3fa9932640 1 -- start start 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.284+0000 7f3fa9932640 1 librados: setting wanted keys 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.284+0000 7f3fa9932640 1 librados: calling monclient init 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.284+0000 7f3fa9932640 10 monclient: init 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3fa9932640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3fa9932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3fa9932640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa406f050 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3fa9932640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa406f430 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3fa9932640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa4070240 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:39.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3fa2ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3fa406f050 0x7f3fa40adfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51266/0 (socket says 172.21.15.55:51266) 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3fa2ffd640 1 -- 172.21.15.55:0/2821254425 learned_addr learned my addr 172.21.15.55:0/2821254425 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3fa9932640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa41ca180 con 0x7f3fa406f050 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3f8bfff640 10 client.?.objecter ms_handle_connect 0x7f3fa406f050 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.285+0000 7f3f8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa413b810 con 0x7f3fa406f430 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 client.?.objecter ms_handle_connect 0x7f3fa4070240 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa413b9a0 con 0x7f3fa4070240 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3fa9932640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3fa9932640 10 monclient(hunting): authenticate will time out at 1152.565727s 2024-03-29T11:45:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353621956 0 0) 0x7f3fa413b9a0 con 0x7f3fa4070240 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f90004eb0 con 0x7f3fa4070240 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898387420 0 0) 0x7f3fa41ca180 con 0x7f3fa406f050 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.286+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa413b9a0 con 0x7f3fa406f050 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 client.?.objecter ms_handle_connect 0x7f3fa406f430 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2038763443 0 0) 0x7f3fa413b9a0 con 0x7f3fa406f050 2024-03-29T11:45:39.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f90005490 con 0x7f3fa406f050 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3f98001970 con 0x7f3fa406f050 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient(hunting): dump: 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:39.300 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 590233244 0 0) 0x7f3f90004eb0 con 0x7f3fa4070240 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f90005b10 con 0x7f3fa4070240 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3f9c001cb0 con 0x7f3fa4070240 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient(hunting): dump: 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:39.301 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1241825832 0 0) 0x7f3f90005490 con 0x7f3fa406f050 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 >> v1:172.21.15.55:6790/0 conn(0x7f3fa4070240 legacy=0x7f3fa413b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 >> v1:172.21.15.171:6789/0 conn(0x7f3fa406f430 legacy=0x7f3fa40ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 monclient: found mon.a 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:39.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa413bb30 con 0x7f3fa406f050 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient: _finish_auth 0 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3f8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3fa9932640 5 monclient: authenticate success, global_id 4350 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3fa9932640 10 monclient: _renew_subs 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3fa9932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa413bd00 con 0x7f3fa406f050 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3fa9932640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3fa9932640 10 monclient: _renew_subs 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3fa9932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa40ab550 con 0x7f3fa406f050 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.287+0000 7f3fa9932640 1 librados: init done 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.289+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f980035f0 con 0x7f3fa406f050 2024-03-29T11:45:39.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.289+0000 7f3f8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.289+0000 7f3fa0ff9640 4 set_mon_vals no callback set 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.289+0000 7f3fa9932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.289+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3f98001970 con 0x7f3fa406f050 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.289+0000 7f3fa9932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.290+0000 7f3fa9932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.290+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f84005080 con 0x7f3fa406f050 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.290+0000 7f3f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.290+0000 7f3f8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.290+0000 7f3f8bfff640 10 monclient: dump: 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:39.304 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:39.305 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:39.305 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:39.305 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:39.305 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:39.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.291+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f3f98003760 con 0x7f3fa406f050 2024-03-29T11:45:39.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.292+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f3f98084580 con 0x7f3fa406f050 2024-03-29T11:45:39.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.292+0000 7f3f8bfff640 10 client.4350.objecter ms_dispatch 0x7f3fa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:39.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.292+0000 7f3f8bfff640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:39.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.292+0000 7f3f8bfff640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:39.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.292+0000 7f3f8bfff640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-29T11:45:39.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.292+0000 7f3fa27fc640 10 monclient: get_auth_request con 0x7f3f9006e390 auth_method 0 2024-03-29T11:45:39.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.292+0000 7f3f8bfff640 10 client.4350.objecter ms_handle_connect 0x7f3f9006e390 2024-03-29T11:45:39.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.299+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f9804fdd0 con 0x7f3fa406f050 2024-03-29T11:45:39.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.299+0000 7f3f8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:39.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.299+0000 7f3f8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:39.653 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.648+0000 7f3fa9932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:39.653 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.648+0000 7f3fa9932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:39.653 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.648+0000 7f3fa9932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:45:39.653 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.648+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f84005330 con 0x7f3fa406f050 2024-03-29T11:45:39.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.649+0000 7f3f8bfff640 1 -- 172.21.15.55:0/2821254425 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f3f98051ed0 con 0x7f3fa406f050 2024-03-29T11:45:39.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.649+0000 7f3f8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:39.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.649+0000 7f3f8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:39.657 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:45:39.658 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:39.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 10 librados: watch_flush enter 2024-03-29T11:45:39.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 10 librados: watch_flush exit 2024-03-29T11:45:39.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-29T11:45:39.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 10 client.4350.objecter successfully canceled tick 2024-03-29T11:45:39.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 >> v1:172.21.15.171:6816/1463197163 conn(0x7f3f9006e390 legacy=0x7f3f90070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:39.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 10 monclient: shutdown 2024-03-29T11:45:39.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:39.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 >> v1:172.21.15.55:6789/0 conn(0x7f3fa406f050 legacy=0x7f3fa40adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:39.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 shutdown_connections 2024-03-29T11:45:39.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 >> 172.21.15.55:0/2821254425 conn(0x7f3fa40b4930 msgr2=0x7f3fa40ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:39.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 shutdown_connections 2024-03-29T11:45:39.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.656+0000 7f3fa9932640 1 -- 172.21.15.55:0/2821254425 wait complete. 2024-03-29T11:45:39.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:39.657+0000 7f3fa9932640 1 librados: shutdown 2024-03-29T11:45:39.677 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:46.484 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:46.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.620+0000 7f01b723e640 10 monclient: get_monmap_and_config 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.620+0000 7f01b723e640 10 monclient: build_initial_monmap 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.621+0000 7f01b723e640 10 monclient: monmap: 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:45:46.621153+0000 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:45:46.621153+0000 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:45:46.625 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.622+0000 7f01b723e640 1 Processor -- start 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.622+0000 7f01b723e640 1 -- start start 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.622+0000 7f01b723e640 10 monclient: init 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.622+0000 7f01b723e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.622+0000 7f01b723e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.622+0000 7f01b723e640 10 monclient(hunting): picked mon.noname-c con 0x7f01b01457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.622+0000 7f01b723e640 10 monclient(hunting): picked mon.noname-a con 0x7f01b013f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01b723e640 10 monclient(hunting): picked mon.noname-b con 0x7f01b00a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01b723e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b00aaef0 con 0x7f01b013f650 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01b57db640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f01b00a9d70 0x7f01b014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:54966/0 (socket says 172.21.15.55:54966) 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01b723e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b0145bd0 con 0x7f01b01457f0 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01b57db640 1 -- 172.21.15.55:0/3448317025 learned_addr learned my addr 172.21.15.55:0/3448317025 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01b723e640 1 -- 172.21.15.55:0/3448317025 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b0145eb0 con 0x7f01b00a9d70 2024-03-29T11:45:46.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01b723e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01b723e640 10 monclient(hunting): authenticate will time out at 1159.901990s 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01af7fe640 1 -- 172.21.15.55:0/3448317025 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305965895 0 0) 0x7f01b0145eb0 con 0x7f01b00a9d70 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01af7fe640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01af7fe640 1 -- 172.21.15.55:0/3448317025 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0198003520 con 0x7f01b00a9d70 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01af7fe640 1 -- 172.21.15.55:0/3448317025 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393271709 0 0) 0x7f0198003520 con 0x7f01b00a9d70 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01af7fe640 1 -- 172.21.15.55:0/3448317025 >> v1:172.21.15.171:6789/0 conn(0x7f01b01457f0 legacy=0x7f01b013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:46.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.623+0000 7f01af7fe640 1 -- 172.21.15.55:0/3448317025 >> v1:172.21.15.55:6789/0 conn(0x7f01b013f650 legacy=0x7f01b00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01af7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01af7fe640 1 monclient: found mon.noname-b 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01af7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01af7fe640 1 -- 172.21.15.55:0/3448317025 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0146340 con 0x7f01b00a9d70 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01af7fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01af7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01af7fe640 1 -- 172.21.15.55:0/3448317025 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f01a8001a10 con 0x7f01b00a9d70 2024-03-29T11:45:46.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.624+0000 7f01b723e640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01b723e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01af7fe640 10 monclient: dump: 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01b723e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:46.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01af7fe640 1 -- 172.21.15.55:0/3448317025 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01a8001c70 con 0x7f01b00a9d70 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01af7fe640 1 -- 172.21.15.55:0/3448317025 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f01a80020c0 con 0x7f01b00a9d70 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01b723e640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01b723e640 4 set_mon_vals no callback set 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.625+0000 7f01af7fe640 10 monclient: dump: 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:46.634 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.626+0000 7f01b723e640 10 monclient: shutdown 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.626+0000 7f01b723e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.626+0000 7f01b723e640 1 -- 172.21.15.55:0/3448317025 >> v1:172.21.15.55:6790/0 conn(0x7f01b00a9d70 legacy=0x7f01b014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.626+0000 7f01b723e640 1 -- 172.21.15.55:0/3448317025 shutdown_connections 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.626+0000 7f01b723e640 1 -- 172.21.15.55:0/3448317025 >> 172.21.15.55:0/3448317025 conn(0x7f01b0066a00 msgr2=0x7f01b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.627+0000 7f01b723e640 1 -- 172.21.15.55:0/3448317025 shutdown_connections 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.627+0000 7f01b723e640 1 -- 172.21.15.55:0/3448317025 wait complete. 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.627+0000 7f01b723e640 10 monclient: build_initial_monmap 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.627+0000 7f01b723e640 10 monclient: monmap: 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:46.635 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.627+0000 7f01b723e640 1 librados: starting msgr at 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.627+0000 7f01b723e640 1 librados: starting objecter 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.628+0000 7f01b723e640 1 Processor -- start 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.628+0000 7f01b723e640 1 -- start start 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.628+0000 7f01b723e640 1 librados: setting wanted keys 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.628+0000 7f01b723e640 1 librados: calling monclient init 2024-03-29T11:45:46.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.628+0000 7f01b723e640 10 monclient: init 2024-03-29T11:45:46.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:46.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 10 monclient(hunting): picked mon.noname-c con 0x7f01b00a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b4fda640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f01b00a9d70 0x7f01b013b4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:55228/0 (socket says 172.21.15.55:55228) 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 10 monclient(hunting): picked mon.noname-a con 0x7f01b013f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 10 monclient(hunting): picked mon.noname-b con 0x7f01b01457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b013b100 con 0x7f01b013f650 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b0139550 con 0x7f01b01457f0 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b00b1770 con 0x7f01b00a9d70 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b723e640 10 monclient(hunting): authenticate will time out at 1159.909125s 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01b4fda640 1 -- 172.21.15.55:0/2517090321 learned_addr learned my addr 172.21.15.55:0/2517090321 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01ad7fa640 10 client.?.objecter ms_handle_connect 0x7f01b00a9d70 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757544032 0 0) 0x7f01b00b1770 con 0x7f01b00a9d70 2024-03-29T11:45:46.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01ad7fa640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f019c004eb0 con 0x7f01b00a9d70 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.630+0000 7f01ad7fa640 10 client.?.objecter ms_handle_connect 0x7f01b01457f0 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174285073 0 0) 0x7f01b0139550 con 0x7f01b01457f0 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b00b1770 con 0x7f01b01457f0 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3969627631 0 0) 0x7f019c004eb0 con 0x7f01b00a9d70 2024-03-29T11:45:46.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01b0139550 con 0x7f01b00a9d70 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f01a0003f60 con 0x7f01b00a9d70 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.631+0000 7f01ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.632+0000 7f01ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.632+0000 7f01ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:46.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.632+0000 7f01ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.632+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2284855256 0 0) 0x7f01b00b1770 con 0x7f01b01457f0 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.632+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f019c004eb0 con 0x7f01b01457f0 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.632+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 663243515 0 0) 0x7f01b0139550 con 0x7f01b00a9d70 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.632+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 >> v1:172.21.15.171:6789/0 conn(0x7f01b01457f0 legacy=0x7f01b01d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.632+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 >> v1:172.21.15.55:6789/0 conn(0x7f01b013f650 legacy=0x7f01b0138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 1 monclient: found mon.c 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0139a70 con 0x7f01b00a9d70 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 10 monclient: _finish_auth 0 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01b723e640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:45:46.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01b723e640 10 monclient: _renew_subs 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01b723e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b0139cc0 con 0x7f01b00a9d70 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01b723e640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01b723e640 10 monclient: _renew_subs 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01b723e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b00b0830 con 0x7f01b00a9d70 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01b723e640 1 librados: init done 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01a0004470 con 0x7f01b00a9d70 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f01a0001070 con 0x7f01b00a9d70 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.633+0000 7f01ad7fa640 10 monclient: dump: 2024-03-29T11:45:46.645 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.634+0000 7f01aeffd640 4 set_mon_vals no callback set 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.634+0000 7f01b723e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.635+0000 7f01b723e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.635+0000 7f01b723e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.635+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0190005080 con 0x7f01b00a9d70 2024-03-29T11:45:46.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.636+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f01a00011d0 con 0x7f01b00a9d70 2024-03-29T11:45:46.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.636+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f01a00848e0 con 0x7f01b00a9d70 2024-03-29T11:45:46.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.636+0000 7f01ad7fa640 10 client.4379.objecter ms_dispatch 0x7f01b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:46.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.636+0000 7f01ad7fa640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:46.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.636+0000 7f01ad7fa640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:46.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.637+0000 7f01ad7fa640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:45:46.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.637+0000 7f01affff640 10 monclient: get_auth_request con 0x7f019c06dd80 auth_method 0 2024-03-29T11:45:46.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.637+0000 7f01ad7fa640 10 client.4379.objecter ms_handle_connect 0x7f019c06dd80 2024-03-29T11:45:46.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.643+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01a00501c0 con 0x7f01b00a9d70 2024-03-29T11:45:46.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.643+0000 7f01ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:46.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.643+0000 7f01ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:46.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.966+0000 7f01b723e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:46.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.966+0000 7f01b723e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:46.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.966+0000 7f01b723e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:46.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.966+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0190005330 con 0x7f01b00a9d70 2024-03-29T11:45:46.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.967+0000 7f01ad7fa640 1 -- 172.21.15.55:0/2517090321 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f01a00520f0 con 0x7f01b00a9d70 2024-03-29T11:45:46.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.967+0000 7f01ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:46.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.967+0000 7f01ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:46.972 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:45:46.973 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.972+0000 7f01b723e640 10 librados: watch_flush enter 2024-03-29T11:45:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.972+0000 7f01b723e640 10 librados: watch_flush exit 2024-03-29T11:45:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.972+0000 7f01b723e640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:45:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.972+0000 7f01b723e640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:45:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.973+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 >> v1:172.21.15.171:6816/1463197163 conn(0x7f019c06dd80 legacy=0x7f019c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.973+0000 7f01b723e640 10 monclient: shutdown 2024-03-29T11:45:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.973+0000 7f01b723e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.973+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 >> v1:172.21.15.55:6790/0 conn(0x7f01b00a9d70 legacy=0x7f01b013b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:46.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.973+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 shutdown_connections 2024-03-29T11:45:46.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.973+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 >> 172.21.15.55:0/2517090321 conn(0x7f01b0066a00 msgr2=0x7f01b00b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:46.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.973+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 shutdown_connections 2024-03-29T11:45:46.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.973+0000 7f01b723e640 1 -- 172.21.15.55:0/2517090321 wait complete. 2024-03-29T11:45:46.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:46.973+0000 7f01b723e640 1 librados: shutdown 2024-03-29T11:45:46.993 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:53.800 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:53.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.945+0000 7f1ffa02e640 10 monclient: get_monmap_and_config 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.945+0000 7f1ffa02e640 10 monclient: build_initial_monmap 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.945+0000 7f1ffa02e640 10 monclient: monmap: 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:45:53.946421+0000 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:45:53.946421+0000 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:45:53.950 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:53.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.946+0000 7f1ffa02e640 1 Processor -- start 2024-03-29T11:45:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.946+0000 7f1ffa02e640 1 -- start start 2024-03-29T11:45:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.946+0000 7f1ffa02e640 10 monclient: init 2024-03-29T11:45:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff41457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 10 monclient(hunting): picked mon.noname-b con 0x7f1ff413f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 10 monclient(hunting): picked mon.noname-c con 0x7f1ff40a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40aaef0 con 0x7f1ff41457f0 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff4145bd0 con 0x7f1ff40a9d70 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff4145eb0 con 0x7f1ff413f650 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.947+0000 7f1ffa02e640 10 monclient(hunting): authenticate will time out at 1167.226996s 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.949+0000 7f1ff37fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1ff41457f0 0x7f1ff413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58213/0 (socket says 172.21.15.55:58213) 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.949+0000 7f1ff37fe640 1 -- 172.21.15.55:0/2315857649 learned_addr learned my addr 172.21.15.55:0/2315857649 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21420012 0 0) 0x7f1ff40aaef0 con 0x7f1ff41457f0 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:45:53.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc003520 con 0x7f1ff41457f0 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777129329 0 0) 0x7f1ff4145eb0 con 0x7f1ff413f650 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.950+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff40aaef0 con 0x7f1ff413f650 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748644076 0 0) 0x7f1fdc003520 con 0x7f1ff41457f0 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 >> v1:172.21.15.55:6790/0 conn(0x7f1ff413f650 legacy=0x7f1ff40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 >> v1:172.21.15.171:6789/0 conn(0x7f1ff40a9d70 legacy=0x7f1ff414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 1 monclient: found mon.noname-a 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff4146340 con 0x7f1ff41457f0 2024-03-29T11:45:53.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1fec001a10 con 0x7f1ff41457f0 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 10 monclient: dump: 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:53.961 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fec001c30 con 0x7f1ff41457f0 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.951+0000 7f1ff27fc640 1 -- 172.21.15.55:0/2315857649 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1fec001fc0 con 0x7f1ff41457f0 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.952+0000 7f1ff27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.952+0000 7f1ff27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.952+0000 7f1ff27fc640 10 monclient: dump: 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:53.962 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.952+0000 7f1ffa02e640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.952+0000 7f1ffa02e640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.952+0000 7f1ffa02e640 4 set_mon_vals no callback set 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.953+0000 7f1ffa02e640 10 monclient: shutdown 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.953+0000 7f1ffa02e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.953+0000 7f1ffa02e640 1 -- 172.21.15.55:0/2315857649 >> v1:172.21.15.55:6789/0 conn(0x7f1ff41457f0 legacy=0x7f1ff413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.953+0000 7f1ffa02e640 1 -- 172.21.15.55:0/2315857649 shutdown_connections 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.953+0000 7f1ffa02e640 1 -- 172.21.15.55:0/2315857649 >> 172.21.15.55:0/2315857649 conn(0x7f1ff4066a00 msgr2=0x7f1ff4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:53.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.953+0000 7f1ffa02e640 1 -- 172.21.15.55:0/2315857649 shutdown_connections 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.953+0000 7f1ffa02e640 1 -- 172.21.15.55:0/2315857649 wait complete. 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.954+0000 7f1ffa02e640 10 monclient: build_initial_monmap 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.954+0000 7f1ffa02e640 10 monclient: monmap: 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:53.966 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.955+0000 7f1ffa02e640 1 librados: starting msgr at 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.955+0000 7f1ffa02e640 1 librados: starting objecter 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.955+0000 7f1ffa02e640 1 Processor -- start 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.955+0000 7f1ffa02e640 1 -- start start 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.955+0000 7f1ffa02e640 1 librados: setting wanted keys 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.955+0000 7f1ffa02e640 1 librados: calling monclient init 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.955+0000 7f1ffa02e640 10 monclient: init 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff40a9d70 addr v1:172.21.15.55:6789/0 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 10 monclient(hunting): picked mon.noname-c con 0x7f1ff413f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:45:53.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 10 monclient(hunting): picked mon.noname-b con 0x7f1ff41457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40701a0 con 0x7f1ff40a9d70 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff4070480 con 0x7f1ff41457f0 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff406f550 con 0x7f1ff413f650 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.956+0000 7f1ffa02e640 10 monclient(hunting): authenticate will time out at 1167.236442s 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.957+0000 7f1ff2ffd640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f1ff413f650 0x7f1ff413a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:60050/0 (socket says 172.21.15.55:60050) 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.957+0000 7f1ff2ffd640 1 -- 172.21.15.55:0/1591800596 learned_addr learned my addr 172.21.15.55:0/1591800596 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.958+0000 7f1fdbfff640 10 client.?.objecter ms_handle_connect 0x7f1ff413f650 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.958+0000 7f1fdbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.958+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985067774 0 0) 0x7f1ff406f550 con 0x7f1ff413f650 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.958+0000 7f1fdbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.958+0000 7f1fdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.958+0000 7f1fdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.958+0000 7f1fdbfff640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:45:53.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.958+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0004eb0 con 0x7f1ff413f650 2024-03-29T11:45:53.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.961+0000 7f1fdbfff640 10 client.?.objecter ms_handle_connect 0x7f1ff40a9d70 2024-03-29T11:45:53.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.961+0000 7f1fdbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:53.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1689997595 0 0) 0x7f1fe0004eb0 con 0x7f1ff413f650 2024-03-29T11:45:53.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ff406f550 con 0x7f1ff413f650 2024-03-29T11:45:53.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715254481 0 0) 0x7f1ff40701a0 con 0x7f1ff40a9d70 2024-03-29T11:45:53.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:53.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:53.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:53.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0004eb0 con 0x7f1ff40a9d70 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1fe8001a50 con 0x7f1ff413f650 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 monclient(hunting): dump: 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:53.970 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 client.?.objecter ms_handle_connect 0x7f1ff41457f0 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.962+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4069224568 0 0) 0x7f1ff406f550 con 0x7f1ff413f650 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 >> v1:172.21.15.171:6789/0 conn(0x7f1ff41457f0 legacy=0x7f1ff4070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 >> v1:172.21.15.55:6789/0 conn(0x7f1ff40a9d70 legacy=0x7f1ff406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 1 monclient: found mon.c 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff406f120 con 0x7f1ff413f650 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 10 monclient: _finish_auth 0 2024-03-29T11:45:53.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ffa02e640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ffa02e640 10 monclient: _renew_subs 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ffa02e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff413aac0 con 0x7f1ff413f650 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ffa02e640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ffa02e640 10 monclient: _renew_subs 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ffa02e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff40717b0 con 0x7f1ff413f650 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ffa02e640 1 librados: init done 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fe8001d30 con 0x7f1ff413f650 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1fdbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.963+0000 7f1ff1ffb640 4 set_mon_vals no callback set 2024-03-29T11:45:53.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.964+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1fe8003a40 con 0x7f1ff413f650 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.964+0000 7f1fdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.964+0000 7f1fdbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.964+0000 7f1fdbfff640 10 monclient: dump: 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:45:53.973 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.965+0000 7f1ffa02e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.965+0000 7f1ffa02e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.965+0000 7f1ffa02e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.965+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f1fe8003ba0 con 0x7f1ff413f650 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.965+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd4005080 con 0x7f1ff413f650 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.966+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f1fe8084b10 con 0x7f1ff413f650 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.966+0000 7f1ff37fe640 10 monclient: get_auth_request con 0x7f1fe006dd60 auth_method 0 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.966+0000 7f1fdbfff640 10 client.4385.objecter ms_dispatch 0x7f1ff4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.966+0000 7f1fdbfff640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:53.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.966+0000 7f1fdbfff640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:53.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.966+0000 7f1fdbfff640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:45:53.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.966+0000 7f1fdbfff640 10 client.4385.objecter ms_handle_connect 0x7f1fe006dd60 2024-03-29T11:45:53.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.975+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fe80503f0 con 0x7f1ff413f650 2024-03-29T11:45:53.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.975+0000 7f1fdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:53.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:53.975+0000 7f1fdbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:54.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.310+0000 7f1ffa02e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:54.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.311+0000 7f1ffa02e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:54.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.311+0000 7f1ffa02e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:45:54.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.311+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fd4005330 con 0x7f1ff413f650 2024-03-29T11:45:54.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.312+0000 7f1fdbfff640 1 -- 172.21.15.55:0/1591800596 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f1fe8052320 con 0x7f1ff413f650 2024-03-29T11:45:54.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.312+0000 7f1fdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:54.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.312+0000 7f1fdbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:54.319 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:45:54.319 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:54.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.316+0000 7f1ffa02e640 10 librados: watch_flush enter 2024-03-29T11:45:54.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.317+0000 7f1ffa02e640 10 librados: watch_flush exit 2024-03-29T11:45:54.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.317+0000 7f1ffa02e640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:45:54.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.317+0000 7f1ffa02e640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:45:54.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.319+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 >> v1:172.21.15.171:6816/1463197163 conn(0x7f1fe006dd60 legacy=0x7f1fe0070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:54.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.319+0000 7f1ffa02e640 10 monclient: shutdown 2024-03-29T11:45:54.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.319+0000 7f1ffa02e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:54.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.319+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 >> v1:172.21.15.55:6790/0 conn(0x7f1ff413f650 legacy=0x7f1ff413a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:54.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.320+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 shutdown_connections 2024-03-29T11:45:54.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.320+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 >> 172.21.15.55:0/1591800596 conn(0x7f1ff4066a00 msgr2=0x7f1ff40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:54.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.320+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 shutdown_connections 2024-03-29T11:45:54.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.320+0000 7f1ffa02e640 1 -- 172.21.15.55:0/1591800596 wait complete. 2024-03-29T11:45:54.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:45:54.320+0000 7f1ffa02e640 1 librados: shutdown 2024-03-29T11:45:54.342 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:01.148 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:01.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.297+0000 7f40c1a33640 10 monclient: get_monmap_and_config 2024-03-29T11:46:01.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.297+0000 7f40c1a33640 10 monclient: build_initial_monmap 2024-03-29T11:46:01.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.297+0000 7f40c1a33640 10 monclient: monmap: 2024-03-29T11:46:01.303 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:01.303 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:01.304 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:46:01.298845+0000 2024-03-29T11:46:01.304 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:46:01.298845+0000 2024-03-29T11:46:01.304 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:01.304 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:01.304 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:01.304 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:46:01.304 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:46:01.304 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:01.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.298+0000 7f40c1a33640 1 Processor -- start 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.298+0000 7f40c1a33640 1 -- start start 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.298+0000 7f40c1a33640 10 monclient: init 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 10 monclient(hunting): picked mon.noname-a con 0x7f40bc0a9750 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 10 monclient(hunting): picked mon.noname-c con 0x7f40bc149ac0 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 10 monclient(hunting): picked mon.noname-b con 0x7f40bc14fe50 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc0aa4a0 con 0x7f40bc0a9750 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc14a680 con 0x7f40bc149ac0 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc0af290 con 0x7f40bc14fe50 2024-03-29T11:46:01.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:01.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.299+0000 7f40c1a33640 10 monclient(hunting): authenticate will time out at 1174.579372s 2024-03-29T11:46:01.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.303+0000 7f40baffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f40bc0a9750 0x7f40bc1436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:60511/0 (socket says 172.21.15.55:60511) 2024-03-29T11:46:01.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.303+0000 7f40baffd640 1 -- 172.21.15.55:0/472430040 learned_addr learned my addr 172.21.15.55:0/472430040 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:01.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.306+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860312348 0 0) 0x7f40bc0aa4a0 con 0x7f40bc0a9750 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.306+0000 7f40b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.306+0000 7f40b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.306+0000 7f40b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.306+0000 7f40b9ffb640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.306+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a4003520 con 0x7f40bc0a9750 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228639749 0 0) 0x7f40bc14a680 con 0x7f40bc149ac0 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40bc0aa4a0 con 0x7f40bc149ac0 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 324279450 0 0) 0x7f40a4003520 con 0x7f40bc0a9750 2024-03-29T11:46:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 >> v1:172.21.15.55:6790/0 conn(0x7f40bc14fe50 legacy=0x7f40bc152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 >> v1:172.21.15.171:6789/0 conn(0x7f40bc149ac0 legacy=0x7f40bc149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 monclient: found mon.noname-a 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc0afb40 con 0x7f40bc0a9750 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40c1a33640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40c1a33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f40b4001a10 con 0x7f40bc0a9750 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:01.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: dump: 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40c1a33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40b4001c30 con 0x7f40bc0a9750 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:01.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40c1a33640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 1 -- 172.21.15.55:0/472430040 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f40b4001fc0 con 0x7f40bc0a9750 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40c1a33640 4 set_mon_vals no callback set 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.307+0000 7f40b9ffb640 10 monclient: dump: 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:01.318 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.308+0000 7f40c1a33640 10 monclient: shutdown 2024-03-29T11:46:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.308+0000 7f40c1a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.308+0000 7f40c1a33640 1 -- 172.21.15.55:0/472430040 >> v1:172.21.15.55:6789/0 conn(0x7f40bc0a9750 legacy=0x7f40bc1436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.308+0000 7f40c1a33640 1 -- 172.21.15.55:0/472430040 shutdown_connections 2024-03-29T11:46:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.308+0000 7f40c1a33640 1 -- 172.21.15.55:0/472430040 >> 172.21.15.55:0/472430040 conn(0x7f40bc0b35e0 msgr2=0x7f40bc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.309+0000 7f40c1a33640 1 -- 172.21.15.55:0/472430040 shutdown_connections 2024-03-29T11:46:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.309+0000 7f40c1a33640 1 -- 172.21.15.55:0/472430040 wait complete. 2024-03-29T11:46:01.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.310+0000 7f40c1a33640 10 monclient: build_initial_monmap 2024-03-29T11:46:01.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.310+0000 7f40c1a33640 10 monclient: monmap: 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.310+0000 7f40c1a33640 1 librados: starting msgr at 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.310+0000 7f40c1a33640 1 librados: starting objecter 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.310+0000 7f40c1a33640 1 Processor -- start 2024-03-29T11:46:01.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.311+0000 7f40c1a33640 1 -- start start 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.311+0000 7f40c1a33640 1 librados: setting wanted keys 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.311+0000 7f40c1a33640 1 librados: calling monclient init 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.311+0000 7f40c1a33640 10 monclient: init 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.312+0000 7f40c1a33640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.312+0000 7f40c1a33640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.315+0000 7f40c1a33640 10 monclient(hunting): picked mon.noname-b con 0x7f40bc0a9750 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.315+0000 7f40baffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f40bc0a9750 0x7f40bc1e7710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:59499/0 (socket says 172.21.15.55:59499) 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.315+0000 7f40c1a33640 10 monclient(hunting): picked mon.noname-a con 0x7f40bc149ac0 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.315+0000 7f40baffd640 1 -- 172.21.15.55:0/22200526 learned_addr learned my addr 172.21.15.55:0/22200526 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.316+0000 7f40c1a33640 10 monclient(hunting): picked mon.noname-c con 0x7f40bc14fe50 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.316+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc0725f0 con 0x7f40bc149ac0 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.316+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc070f20 con 0x7f40bc0a9750 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.316+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc071420 con 0x7f40bc14fe50 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.316+0000 7f40c1a33640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.316+0000 7f40c1a33640 10 monclient(hunting): authenticate will time out at 1174.595689s 2024-03-29T11:46:01.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.319+0000 7f40a3fff640 10 client.?.objecter ms_handle_connect 0x7f40bc0a9750 2024-03-29T11:46:01.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.319+0000 7f40a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:01.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.319+0000 7f40a3fff640 10 client.?.objecter ms_handle_connect 0x7f40bc14fe50 2024-03-29T11:46:01.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.319+0000 7f40a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:01.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.319+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937712464 0 0) 0x7f40bc070f20 con 0x7f40bc0a9750 2024-03-29T11:46:01.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:46:01.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a8004eb0 con 0x7f40bc0a9750 2024-03-29T11:46:01.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743899888 0 0) 0x7f40bc071420 con 0x7f40bc14fe50 2024-03-29T11:46:01.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:46:01.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.320+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40bc070f20 con 0x7f40bc14fe50 2024-03-29T11:46:01.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.324+0000 7f40a3fff640 10 client.?.objecter ms_handle_connect 0x7f40bc149ac0 2024-03-29T11:46:01.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.324+0000 7f40a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:01.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.324+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638632811 0 0) 0x7f40bc0725f0 con 0x7f40bc149ac0 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.324+0000 7f40a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.324+0000 7f40a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.324+0000 7f40a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.324+0000 7f40a3fff640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.324+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40bc071420 con 0x7f40bc149ac0 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.324+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3634292447 0 0) 0x7f40bc070f20 con 0x7f40bc14fe50 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40bc0725f0 con 0x7f40bc14fe50 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1958093169 0 0) 0x7f40a8004eb0 con 0x7f40bc0a9750 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40bc070f20 con 0x7f40bc0a9750 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f40ac001830 con 0x7f40bc14fe50 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 10 monclient(hunting): dump: 2024-03-29T11:46:01.332 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f40b4001f40 con 0x7f40bc0a9750 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.325+0000 7f40a3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 10 monclient(hunting): dump: 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:01.333 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1933408969 0 0) 0x7f40bc071420 con 0x7f40bc149ac0 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40a8004eb0 con 0x7f40bc149ac0 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2111427628 0 0) 0x7f40bc0725f0 con 0x7f40bc14fe50 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 >> v1:172.21.15.171:6789/0 conn(0x7f40bc0a9750 legacy=0x7f40bc1e7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 >> v1:172.21.15.55:6789/0 conn(0x7f40bc149ac0 legacy=0x7f40bc1e1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 1 monclient: found mon.c 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc0715b0 con 0x7f40bc14fe50 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 10 monclient: _finish_auth 0 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40c1a33640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40c1a33640 10 monclient: _renew_subs 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40c1a33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc06ff90 con 0x7f40bc14fe50 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40c1a33640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40c1a33640 10 monclient: _renew_subs 2024-03-29T11:46:01.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40c1a33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:01.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc1e8000 con 0x7f40bc14fe50 2024-03-29T11:46:01.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40c1a33640 1 librados: init done 2024-03-29T11:46:01.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40ac0035f0 con 0x7f40bc14fe50 2024-03-29T11:46:01.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:01.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40b8ff9640 4 set_mon_vals no callback set 2024-03-29T11:46:01.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f40ac003980 con 0x7f40bc14fe50 2024-03-29T11:46:01.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.326+0000 7f40a3fff640 10 monclient: dump: 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:01.338 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.328+0000 7f40c1a33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.329+0000 7f40c1a33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.329+0000 7f40c1a33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.329+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409c005080 con 0x7f40bc14fe50 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.329+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f40ac003ae0 con 0x7f40bc14fe50 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.330+0000 7f40baffd640 10 monclient: get_auth_request con 0x7f40a806e1a0 auth_method 0 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.330+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f40ac0836c0 con 0x7f40bc14fe50 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.330+0000 7f40a3fff640 10 client.4391.objecter ms_dispatch 0x7f40bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.330+0000 7f40a3fff640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.330+0000 7f40a3fff640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.330+0000 7f40a3fff640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T11:46:01.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.330+0000 7f40a3fff640 10 client.4391.objecter ms_handle_connect 0x7f40a806e1a0 2024-03-29T11:46:01.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.340+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40ac04efa0 con 0x7f40bc14fe50 2024-03-29T11:46:01.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.340+0000 7f40a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:01.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.340+0000 7f40a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:01.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.669+0000 7f40c1a33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:01.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.669+0000 7f40c1a33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:01.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.669+0000 7f40c1a33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:01.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.669+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f409c005330 con 0x7f40bc14fe50 2024-03-29T11:46:01.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.671+0000 7f40a3fff640 1 -- 172.21.15.55:0/22200526 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f40ac050ed0 con 0x7f40bc14fe50 2024-03-29T11:46:01.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.671+0000 7f40a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:01.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.671+0000 7f40a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:01.676 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:46:01.676 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.674+0000 7f40c1a33640 10 librados: watch_flush enter 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.674+0000 7f40c1a33640 10 librados: watch_flush exit 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 10 client.4391.objecter successfully canceled tick 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 >> v1:172.21.15.171:6816/1463197163 conn(0x7f40a806e1a0 legacy=0x7f40a8070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 10 monclient: shutdown 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 >> v1:172.21.15.55:6790/0 conn(0x7f40bc14fe50 legacy=0x7f40bc1e1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 shutdown_connections 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 >> 172.21.15.55:0/22200526 conn(0x7f40bc0b35e0 msgr2=0x7f40bc0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 shutdown_connections 2024-03-29T11:46:01.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 1 -- 172.21.15.55:0/22200526 wait complete. 2024-03-29T11:46:01.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:01.675+0000 7f40c1a33640 1 librados: shutdown 2024-03-29T11:46:01.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:08.509 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:08.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.659+0000 7f2d95a15640 10 monclient: get_monmap_and_config 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.659+0000 7f2d95a15640 10 monclient: build_initial_monmap 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.659+0000 7f2d95a15640 10 monclient: monmap: 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:46:08.660780+0000 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:46:08.660780+0000 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:46:08.665 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:08.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 1 Processor -- start 2024-03-29T11:46:08.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 1 -- start start 2024-03-29T11:46:08.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 10 monclient: init 2024-03-29T11:46:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:46:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 10 monclient(hunting): picked mon.noname-c con 0x7f2d900b4330 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 10 monclient(hunting): picked mon.noname-b con 0x7f2d901470b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 10 monclient(hunting): picked mon.noname-a con 0x7f2d901439b0 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9006f2a0 con 0x7f2d901439b0 2024-03-29T11:46:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d90146b90 con 0x7f2d900b4330 2024-03-29T11:46:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d900b2ca0 con 0x7f2d901470b0 2024-03-29T11:46:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.661+0000 7f2d95a15640 10 monclient(hunting): authenticate will time out at 1181.941605s 2024-03-29T11:46:08.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.665+0000 7f2d8f7fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2d901439b0 0x7f2d90151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:53771/0 (socket says 172.21.15.55:53771) 2024-03-29T11:46:08.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.665+0000 7f2d8f7fe640 1 -- 172.21.15.55:0/3496522971 learned_addr learned my addr 172.21.15.55:0/3496522971 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526376448 0 0) 0x7f2d9006f2a0 con 0x7f2d901439b0 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d7c003520 con 0x7f2d901439b0 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260588267 0 0) 0x7f2d90146b90 con 0x7f2d900b4330 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:46:08.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d9006f2a0 con 0x7f2d900b4330 2024-03-29T11:46:08.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3686669514 0 0) 0x7f2d7c003520 con 0x7f2d901439b0 2024-03-29T11:46:08.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.666+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 >> v1:172.21.15.55:6790/0 conn(0x7f2d901470b0 legacy=0x7f2d900b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:46:08.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 >> v1:172.21.15.171:6789/0 conn(0x7f2d900b4330 legacy=0x7f2d900b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:08.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9006e0c0 con 0x7f2d901439b0 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2d88001a10 con 0x7f2d901439b0 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.668+0000 7f2d8e7fc640 10 monclient: dump: 2024-03-29T11:46:08.676 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d95a15640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d95a15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d88001b70 con 0x7f2d901439b0 2024-03-29T11:46:08.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d8e7fc640 1 -- 172.21.15.55:0/3496522971 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2d88001f80 con 0x7f2d901439b0 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d8e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d8e7fc640 10 monclient: dump: 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:08.678 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:08.679 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:08.679 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:08.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d95a15640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:08.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.669+0000 7f2d95a15640 4 set_mon_vals no callback set 2024-03-29T11:46:08.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.671+0000 7f2d95a15640 10 monclient: shutdown 2024-03-29T11:46:08.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.671+0000 7f2d95a15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:08.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.671+0000 7f2d95a15640 1 -- 172.21.15.55:0/3496522971 >> v1:172.21.15.55:6789/0 conn(0x7f2d901439b0 legacy=0x7f2d90151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:08.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.671+0000 7f2d95a15640 1 -- 172.21.15.55:0/3496522971 shutdown_connections 2024-03-29T11:46:08.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.671+0000 7f2d95a15640 1 -- 172.21.15.55:0/3496522971 >> 172.21.15.55:0/3496522971 conn(0x7f2d90066a00 msgr2=0x7f2d90066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:08.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.672+0000 7f2d95a15640 1 -- 172.21.15.55:0/3496522971 shutdown_connections 2024-03-29T11:46:08.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.672+0000 7f2d95a15640 1 -- 172.21.15.55:0/3496522971 wait complete. 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.672+0000 7f2d95a15640 10 monclient: build_initial_monmap 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.672+0000 7f2d95a15640 10 monclient: monmap: 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:46:08.682 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.673+0000 7f2d95a15640 1 librados: starting msgr at 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.673+0000 7f2d95a15640 1 librados: starting objecter 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.673+0000 7f2d95a15640 1 Processor -- start 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.673+0000 7f2d95a15640 1 -- start start 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.673+0000 7f2d95a15640 1 librados: setting wanted keys 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.673+0000 7f2d95a15640 1 librados: calling monclient init 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.673+0000 7f2d95a15640 10 monclient: init 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 10 monclient(hunting): picked mon.noname-b con 0x7f2d900b4330 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 10 monclient(hunting): picked mon.noname-a con 0x7f2d901439b0 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 10 monclient(hunting): picked mon.noname-c con 0x7f2d901470b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d901db670 con 0x7f2d901439b0 2024-03-29T11:46:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d901dc2a0 con 0x7f2d900b4330 2024-03-29T11:46:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d901254b0 con 0x7f2d901470b0 2024-03-29T11:46:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.674+0000 7f2d95a15640 10 monclient(hunting): authenticate will time out at 1181.954479s 2024-03-29T11:46:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.679+0000 7f2d87fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2d901439b0 0x7f2d901dbb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:53871/0 (socket says 172.21.15.55:53871) 2024-03-29T11:46:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.679+0000 7f2d87fff640 1 -- 172.21.15.55:0/1569187763 learned_addr learned my addr 172.21.15.55:0/1569187763 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.679+0000 7f2d8cff9640 10 client.?.objecter ms_handle_connect 0x7f2d901439b0 2024-03-29T11:46:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.679+0000 7f2d8cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.679+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617371513 0 0) 0x7f2d901db670 con 0x7f2d901439b0 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d70004eb0 con 0x7f2d901439b0 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3069801265 0 0) 0x7f2d70004eb0 con 0x7f2d901439b0 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d901db670 con 0x7f2d901439b0 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2d80001960 con 0x7f2d901439b0 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 10 monclient(hunting): dump: 2024-03-29T11:46:08.688 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.680+0000 7f2d8cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4236175276 0 0) 0x7f2d901db670 con 0x7f2d901439b0 2024-03-29T11:46:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 >> v1:172.21.15.55:6790/0 conn(0x7f2d901470b0 legacy=0x7f2d90140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 >> v1:172.21.15.171:6789/0 conn(0x7f2d900b4330 legacy=0x7f2d90125c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 1 monclient: found mon.a 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d90122330 con 0x7f2d901439b0 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 10 monclient: _finish_auth 0 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d80001c80 con 0x7f2d901439b0 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2d80003a40 con 0x7f2d901439b0 2024-03-29T11:46:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8cff9640 10 monclient: dump: 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:08.691 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.681+0000 7f2d8dffb640 4 set_mon_vals no callback set 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.682+0000 7f2d95a15640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.682+0000 7f2d95a15640 10 monclient: _renew_subs 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.682+0000 7f2d95a15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.682+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d90122580 con 0x7f2d901439b0 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.682+0000 7f2d95a15640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.682+0000 7f2d95a15640 10 monclient: _renew_subs 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.682+0000 7f2d95a15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.682+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d901250c0 con 0x7f2d901439b0 2024-03-29T11:46:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.682+0000 7f2d95a15640 1 librados: init done 2024-03-29T11:46:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.683+0000 7f2d95a15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.683+0000 7f2d95a15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.683+0000 7f2d95a15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.683+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d901db460 con 0x7f2d901439b0 2024-03-29T11:46:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.686+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f2d80003ba0 con 0x7f2d901439b0 2024-03-29T11:46:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.687+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f2d80084af0 con 0x7f2d901439b0 2024-03-29T11:46:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.687+0000 7f2d8cff9640 10 client.4368.objecter ms_dispatch 0x7f2d90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.687+0000 7f2d8cff9640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:08.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.687+0000 7f2d8cff9640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:08.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.687+0000 7f2d8cff9640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:46:08.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.688+0000 7f2d8effd640 10 monclient: get_auth_request con 0x7f2d7006dc00 auth_method 0 2024-03-29T11:46:08.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.689+0000 7f2d8cff9640 10 client.4368.objecter ms_handle_connect 0x7f2d7006dc00 2024-03-29T11:46:08.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.693+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d800503d0 con 0x7f2d901439b0 2024-03-29T11:46:08.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.693+0000 7f2d8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:08.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:08.693+0000 7f2d8cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:09.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.017+0000 7f2d95a15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:09.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.017+0000 7f2d95a15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:09.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.017+0000 7f2d95a15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:09.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.018+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d90075f10 con 0x7f2d901439b0 2024-03-29T11:46:09.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.019+0000 7f2d8cff9640 1 -- 172.21.15.55:0/1569187763 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f2d80052300 con 0x7f2d901439b0 2024-03-29T11:46:09.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.019+0000 7f2d8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:09.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.019+0000 7f2d8cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:09.028 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:46:09.028 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 10 librados: watch_flush enter 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 10 librados: watch_flush exit 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 >> v1:172.21.15.171:6816/1463197163 conn(0x7f2d7006dc00 legacy=0x7f2d700700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 10 monclient: shutdown 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 >> v1:172.21.15.55:6789/0 conn(0x7f2d901439b0 legacy=0x7f2d901dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 shutdown_connections 2024-03-29T11:46:09.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 >> 172.21.15.55:0/1569187763 conn(0x7f2d90066a00 msgr2=0x7f2d90066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:09.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 shutdown_connections 2024-03-29T11:46:09.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 1 -- 172.21.15.55:0/1569187763 wait complete. 2024-03-29T11:46:09.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:09.024+0000 7f2d95a15640 1 librados: shutdown 2024-03-29T11:46:09.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:15.853 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:16.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.995+0000 7f1f85405640 10 monclient: get_monmap_and_config 2024-03-29T11:46:16.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.995+0000 7f1f85405640 10 monclient: build_initial_monmap 2024-03-29T11:46:16.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.995+0000 7f1f85405640 10 monclient: monmap: 2024-03-29T11:46:16.000 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:16.000 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:16.000 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:46:15.996568+0000 2024-03-29T11:46:16.001 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:46:15.996568+0000 2024-03-29T11:46:16.001 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:16.001 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:16.001 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:16.001 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:46:16.001 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:46:16.001 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:16.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.997+0000 7f1f85405640 1 Processor -- start 2024-03-29T11:46:16.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.998+0000 7f1f85405640 1 -- start start 2024-03-29T11:46:16.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.998+0000 7f1f85405640 10 monclient: init 2024-03-29T11:46:16.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:16.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:46:16.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 10 monclient(hunting): picked mon.noname-c con 0x7f1f800b0770 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:16.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8014a700 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:16.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 10 monclient(hunting): picked mon.noname-b con 0x7f1f800ac5a0 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:16.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800aba70 con 0x7f1f8014a700 2024-03-29T11:46:16.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800aecb0 con 0x7f1f800b0770 2024-03-29T11:46:16.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800af150 con 0x7f1f800ac5a0 2024-03-29T11:46:16.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:16.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:15.999+0000 7f1f85405640 10 monclient(hunting): authenticate will time out at 1189.278959s 2024-03-29T11:46:16.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.003+0000 7f1f7e7fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1f8014a700 0x7f1f800abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:35861/0 (socket says 172.21.15.55:35861) 2024-03-29T11:46:16.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.003+0000 7f1f7e7fc640 1 -- 172.21.15.55:0/3899005580 learned_addr learned my addr 172.21.15.55:0/3899005580 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:16.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.004+0000 7f1f7dffb640 1 -- 172.21.15.55:0/3899005580 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376851727 0 0) 0x7f1f800aba70 con 0x7f1f8014a700 2024-03-29T11:46:16.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.004+0000 7f1f7dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:46:16.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.004+0000 7f1f7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:16.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.004+0000 7f1f7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:16.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.004+0000 7f1f7dffb640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:46:16.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.004+0000 7f1f7dffb640 1 -- 172.21.15.55:0/3899005580 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f68003520 con 0x7f1f8014a700 2024-03-29T11:46:16.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.004+0000 7f1f7dffb640 1 -- 172.21.15.55:0/3899005580 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4018751392 0 0) 0x7f1f68003520 con 0x7f1f8014a700 2024-03-29T11:46:16.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.004+0000 7f1f7dffb640 1 -- 172.21.15.55:0/3899005580 >> v1:172.21.15.55:6790/0 conn(0x7f1f800ac5a0 legacy=0x7f1f80151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 1 -- 172.21.15.55:0/3899005580 >> v1:172.21.15.171:6789/0 conn(0x7f1f800b0770 legacy=0x7f1f80149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 1 monclient: found mon.noname-a 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 1 -- 172.21.15.55:0/3899005580 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f800ac9d0 con 0x7f1f8014a700 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 10 monclient: _finish_auth 0 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f85405640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f85405640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 1 -- 172.21.15.55:0/3899005580 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1f74001690 con 0x7f1f8014a700 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:16.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.006+0000 7f1f7dffb640 10 monclient: dump: 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:16.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.007+0000 7f1f85405640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.008+0000 7f1f7dffb640 1 -- 172.21.15.55:0/3899005580 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f74001920 con 0x7f1f8014a700 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.008+0000 7f1f7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.008+0000 7f1f7dffb640 1 -- 172.21.15.55:0/3899005580 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1f74001cb0 con 0x7f1f8014a700 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.008+0000 7f1f7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.008+0000 7f1f7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.008+0000 7f1f7dffb640 10 monclient: dump: 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:16.014 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:16.015 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:16.015 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:16.015 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:16.015 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:16.015 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:16.015 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:16.015 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.008+0000 7f1f85405640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:16.015 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.008+0000 7f1f85405640 4 set_mon_vals no callback set 2024-03-29T11:46:16.015 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.009+0000 7f1f85405640 10 monclient: shutdown 2024-03-29T11:46:16.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.009+0000 7f1f85405640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:16.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.009+0000 7f1f85405640 1 -- 172.21.15.55:0/3899005580 >> v1:172.21.15.55:6789/0 conn(0x7f1f8014a700 legacy=0x7f1f800abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:16.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.012+0000 7f1f85405640 1 -- 172.21.15.55:0/3899005580 shutdown_connections 2024-03-29T11:46:16.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.012+0000 7f1f85405640 1 -- 172.21.15.55:0/3899005580 >> 172.21.15.55:0/3899005580 conn(0x7f1f8006b660 msgr2=0x7f1f8006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:16.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.012+0000 7f1f85405640 1 -- 172.21.15.55:0/3899005580 shutdown_connections 2024-03-29T11:46:16.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.013+0000 7f1f85405640 1 -- 172.21.15.55:0/3899005580 wait complete. 2024-03-29T11:46:16.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.013+0000 7f1f85405640 10 monclient: build_initial_monmap 2024-03-29T11:46:16.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.013+0000 7f1f85405640 10 monclient: monmap: 2024-03-29T11:46:16.018 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:16.018 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:16.019 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:46:16.019 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:46:16.019 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:16.019 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:16.019 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:16.019 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:46:16.019 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:46:16.019 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:16.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.014+0000 7f1f85405640 1 librados: starting msgr at 2024-03-29T11:46:16.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.014+0000 7f1f85405640 1 librados: starting objecter 2024-03-29T11:46:16.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.014+0000 7f1f85405640 1 Processor -- start 2024-03-29T11:46:16.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 1 -- start start 2024-03-29T11:46:16.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 1 librados: setting wanted keys 2024-03-29T11:46:16.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 1 librados: calling monclient init 2024-03-29T11:46:16.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 10 monclient: init 2024-03-29T11:46:16.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:16.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:46:16.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 10 monclient(hunting): picked mon.noname-b con 0x7f1f800ac5a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:16.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 10 monclient(hunting): picked mon.noname-a con 0x7f1f800b0770 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:16.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 10 monclient(hunting): picked mon.noname-c con 0x7f1f8014a700 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:16.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f801db360 con 0x7f1f800b0770 2024-03-29T11:46:16.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f801239d0 con 0x7f1f800ac5a0 2024-03-29T11:46:16.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f80145a80 con 0x7f1f8014a700 2024-03-29T11:46:16.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:16.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.015+0000 7f1f85405640 10 monclient(hunting): authenticate will time out at 1189.295436s 2024-03-29T11:46:16.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.020+0000 7f1f7effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f1f800ac5a0 0x7f1f801db880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:42489/0 (socket says 172.21.15.55:42489) 2024-03-29T11:46:16.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.021+0000 7f1f7effd640 1 -- 172.21.15.55:0/3606352122 learned_addr learned my addr 172.21.15.55:0/3606352122 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:16.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.021+0000 7f1f677fe640 10 client.?.objecter ms_handle_connect 0x7f1f800b0770 2024-03-29T11:46:16.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.021+0000 7f1f677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:16.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.021+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156409426 0 0) 0x7f1f801db360 con 0x7f1f800b0770 2024-03-29T11:46:16.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.022+0000 7f1f677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:46:16.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.022+0000 7f1f677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:16.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.022+0000 7f1f677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:16.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.022+0000 7f1f677fe640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:46:16.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.022+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c004eb0 con 0x7f1f800b0770 2024-03-29T11:46:16.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.022+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 919920136 0 0) 0x7f1f6c004eb0 con 0x7f1f800b0770 2024-03-29T11:46:16.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.022+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f801db360 con 0x7f1f800b0770 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.022+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1f74001610 con 0x7f1f800b0770 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 10 monclient(hunting): dump: 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:16.030 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2244141890 0 0) 0x7f1f801db360 con 0x7f1f800b0770 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 >> v1:172.21.15.55:6790/0 conn(0x7f1f8014a700 legacy=0x7f1f80144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 >> v1:172.21.15.171:6789/0 conn(0x7f1f800ac5a0 legacy=0x7f1f801db880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 1 monclient: found mon.a 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f80145c10 con 0x7f1f800b0770 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.023+0000 7f1f677fe640 10 monclient: _finish_auth 0 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:46:16.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f85405640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f85405640 10 monclient: _renew_subs 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f85405640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f801dbf90 con 0x7f1f800b0770 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f85405640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f85405640 10 monclient: _renew_subs 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f85405640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f80126fd0 con 0x7f1f800b0770 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f85405640 1 librados: init done 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f74004a90 con 0x7f1f800b0770 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1f74004da0 con 0x7f1f800b0770 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.024+0000 7f1f677fe640 10 monclient: dump: 2024-03-29T11:46:16.032 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:16.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.025+0000 7f1f7d7fa640 4 set_mon_vals no callback set 2024-03-29T11:46:16.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f1f7401ad70 con 0x7f1f800b0770 2024-03-29T11:46:16.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f1f74084b60 con 0x7f1f800b0770 2024-03-29T11:46:16.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f677fe640 10 client.4374.objecter ms_dispatch 0x7f1f80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:16.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f677fe640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:16.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f677fe640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:16.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f677fe640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T11:46:16.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f85405640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:16.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f85405640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:16.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f85405640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:16.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.027+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f60005080 con 0x7f1f800b0770 2024-03-29T11:46:16.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.030+0000 7f1f7effd640 10 monclient: get_auth_request con 0x7f1f6c06dd20 auth_method 0 2024-03-29T11:46:16.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.031+0000 7f1f677fe640 10 client.4374.objecter ms_handle_connect 0x7f1f6c06dd20 2024-03-29T11:46:16.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.036+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f74050440 con 0x7f1f800b0770 2024-03-29T11:46:16.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.036+0000 7f1f677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:16.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.036+0000 7f1f677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:16.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.373+0000 7f1f85405640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:16.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.373+0000 7f1f85405640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:16.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.373+0000 7f1f85405640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:16.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.373+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f60005330 con 0x7f1f800b0770 2024-03-29T11:46:16.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.378+0000 7f1f677fe640 1 -- 172.21.15.55:0/3606352122 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f1f74052370 con 0x7f1f800b0770 2024-03-29T11:46:16.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.378+0000 7f1f677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:16.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.378+0000 7f1f677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:16.383 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:46:16.383 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:16.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.381+0000 7f1f85405640 10 librados: watch_flush enter 2024-03-29T11:46:16.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.381+0000 7f1f85405640 10 librados: watch_flush exit 2024-03-29T11:46:16.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.381+0000 7f1f85405640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T11:46:16.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.381+0000 7f1f85405640 10 client.4374.objecter successfully canceled tick 2024-03-29T11:46:16.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.381+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 >> v1:172.21.15.171:6816/1463197163 conn(0x7f1f6c06dd20 legacy=0x7f1f6c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:16.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.381+0000 7f1f85405640 10 monclient: shutdown 2024-03-29T11:46:16.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.381+0000 7f1f85405640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:16.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.382+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 >> v1:172.21.15.55:6789/0 conn(0x7f1f800b0770 legacy=0x7f1f801266e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.382+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 shutdown_connections 2024-03-29T11:46:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.382+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 >> 172.21.15.55:0/3606352122 conn(0x7f1f8006b660 msgr2=0x7f1f800abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.382+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 shutdown_connections 2024-03-29T11:46:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.382+0000 7f1f85405640 1 -- 172.21.15.55:0/3606352122 wait complete. 2024-03-29T11:46:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:16.382+0000 7f1f85405640 1 librados: shutdown 2024-03-29T11:46:16.404 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:23.211 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:23.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.361+0000 7fe200d64640 10 monclient: get_monmap_and_config 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.361+0000 7fe200d64640 10 monclient: build_initial_monmap 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.361+0000 7fe200d64640 10 monclient: monmap: 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:46:23.362516+0000 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:46:23.362516+0000 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.363+0000 7fe200d64640 1 Processor -- start 2024-03-29T11:46:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.363+0000 7fe200d64640 1 -- start start 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.363+0000 7fe200d64640 10 monclient: init 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 10 monclient(hunting): picked mon.noname-b con 0x7fe1f40e2d10 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 10 monclient(hunting): picked mon.noname-a con 0x7fe1f402b090 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 10 monclient(hunting): picked mon.noname-c con 0x7fe1f402c7f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f402ac40 con 0x7fe1f402b090 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f402a430 con 0x7fe1f402c7f0 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f402a5c0 con 0x7fe1f40e2d10 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:23.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.364+0000 7fe200d64640 10 monclient(hunting): authenticate will time out at 1196.644603s 2024-03-29T11:46:23.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.369+0000 7fe1fb7fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fe1f40e2d10 0x7fe1f40e8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:55949/0 (socket says 172.21.15.55:55949) 2024-03-29T11:46:23.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.369+0000 7fe1fb7fe640 1 -- 172.21.15.55:0/3580653144 learned_addr learned my addr 172.21.15.55:0/3580653144 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:23.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.369+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065717580 0 0) 0x7fe1f402a5c0 con 0x7fe1f40e2d10 2024-03-29T11:46:23.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.369+0000 7fe1fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:23.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.369+0000 7fe1fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.369+0000 7fe1fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.369+0000 7fe1fa7fc640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.369+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ec005730 con 0x7fe1f40e2d10 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785353890 0 0) 0x7fe1f402ac40 con 0x7fe1f402b090 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ec006ab0 con 0x7fe1f402b090 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986283988 0 0) 0x7fe1ec006ab0 con 0x7fe1f402b090 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 >> v1:172.21.15.55:6790/0 conn(0x7fe1f40e2d10 legacy=0x7fe1f40e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 >> v1:172.21.15.171:6789/0 conn(0x7fe1f402c7f0 legacy=0x7fe1f40ebc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:46:23.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 1 monclient: found mon.noname-a 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f4006940 con 0x7fe1f402b090 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.370+0000 7fe1fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe1fc04f730 con 0x7fe1f402b090 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe1fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe1fa7fc640 10 monclient: dump: 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:23.380 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe200d64640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe200d64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1fc0008d0 con 0x7fe1f402b090 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe1fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe200d64640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.371+0000 7fe200d64640 4 set_mon_vals no callback set 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.372+0000 7fe1fa7fc640 1 -- 172.21.15.55:0/3580653144 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe1fc04f730 con 0x7fe1f402b090 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.372+0000 7fe1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.372+0000 7fe1fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.372+0000 7fe1fa7fc640 10 monclient: dump: 2024-03-29T11:46:23.381 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.372+0000 7fe200d64640 10 monclient: shutdown 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.372+0000 7fe200d64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.372+0000 7fe200d64640 1 -- 172.21.15.55:0/3580653144 >> v1:172.21.15.55:6789/0 conn(0x7fe1f402b090 legacy=0x7fe1f4029c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.373+0000 7fe200d64640 1 -- 172.21.15.55:0/3580653144 shutdown_connections 2024-03-29T11:46:23.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.373+0000 7fe200d64640 1 -- 172.21.15.55:0/3580653144 >> 172.21.15.55:0/3580653144 conn(0x7fe1f40058f0 msgr2=0x7fe1f4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:23.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.373+0000 7fe200d64640 1 -- 172.21.15.55:0/3580653144 shutdown_connections 2024-03-29T11:46:23.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.373+0000 7fe200d64640 1 -- 172.21.15.55:0/3580653144 wait complete. 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.374+0000 7fe200d64640 10 monclient: build_initial_monmap 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.374+0000 7fe200d64640 10 monclient: monmap: 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.375+0000 7fe200d64640 1 librados: starting msgr at 2024-03-29T11:46:23.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.375+0000 7fe200d64640 1 librados: starting objecter 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.375+0000 7fe200d64640 1 Processor -- start 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.375+0000 7fe200d64640 1 -- start start 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.376+0000 7fe200d64640 1 librados: setting wanted keys 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.376+0000 7fe200d64640 1 librados: calling monclient init 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.376+0000 7fe200d64640 10 monclient: init 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 10 monclient(hunting): picked mon.noname-b con 0x7fe1f402b090 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 10 monclient(hunting): picked mon.noname-c con 0x7fe1f402c7f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 10 monclient(hunting): picked mon.noname-a con 0x7fe1f40e2d10 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f402e530 con 0x7fe1f40e2d10 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f402e7a0 con 0x7fe1f402b090 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f40e5dd0 con 0x7fe1f402c7f0 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:23.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.377+0000 7fe200d64640 10 monclient(hunting): authenticate will time out at 1196.657545s 2024-03-29T11:46:23.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.380+0000 7fe1fbfff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fe1f40e2d10 0x7fe1f40e3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42903/0 (socket says 172.21.15.55:42903) 2024-03-29T11:46:23.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.380+0000 7fe1fbfff640 1 -- 172.21.15.55:0/2591366103 learned_addr learned my addr 172.21.15.55:0/2591366103 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:23.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 10 client.?.objecter ms_handle_connect 0x7fe1f40e2d10 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439766564 0 0) 0x7fe1f402e530 con 0x7fe1f40e2d10 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e0004eb0 con 0x7fe1f40e2d10 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 654337524 0 0) 0x7fe1e0004eb0 con 0x7fe1f40e2d10 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.381+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1f402e530 con 0x7fe1f40e2d10 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.382+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe1f00019b0 con 0x7fe1f40e2d10 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.382+0000 7fe1e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.382+0000 7fe1e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.382+0000 7fe1e7fff640 10 monclient(hunting): dump: 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:23.389 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:23.390 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:23.390 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:23.390 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:23.390 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:23.390 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:23.390 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:23.390 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:23.390 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:23.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.382+0000 7fe1e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:23.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.385+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2141680155 0 0) 0x7fe1f402e530 con 0x7fe1f40e2d10 2024-03-29T11:46:23.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.385+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 >> v1:172.21.15.55:6790/0 conn(0x7fe1f402c7f0 legacy=0x7fe1f40e18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:23.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.385+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 >> v1:172.21.15.171:6789/0 conn(0x7fe1f402b090 legacy=0x7fe1f402ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:23.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.385+0000 7fe1e7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:46:23.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.385+0000 7fe1e7fff640 1 monclient: found mon.a 2024-03-29T11:46:23.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.385+0000 7fe1e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:23.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe1e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f40e5f60 con 0x7fe1f40e2d10 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe1e7fff640 10 monclient: _finish_auth 0 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe1e7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe1e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe1e7fff640 10 client.?.objecter ms_handle_connect 0x7fe1f402b090 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe1e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe200d64640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe200d64640 10 monclient: _renew_subs 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe200d64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f40e1ff0 con 0x7fe1f40e2d10 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe200d64640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe200d64640 10 monclient: _renew_subs 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe200d64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:23.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f40e3c70 con 0x7fe1f40e2d10 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.386+0000 7fe200d64640 1 librados: init done 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.387+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1f0001cb0 con 0x7fe1f40e2d10 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.387+0000 7fe1e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.387+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe1f0003a60 con 0x7fe1f40e2d10 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.387+0000 7fe1e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.387+0000 7fe1e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.387+0000 7fe1e7fff640 10 monclient: dump: 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:23.395 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:23.396 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:23.396 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:23.396 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:23.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.387+0000 7fe1f9ffb640 4 set_mon_vals no callback set 2024-03-29T11:46:23.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.389+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fe1f0003bc0 con 0x7fe1f40e2d10 2024-03-29T11:46:23.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.389+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fe1f0084b60 con 0x7fe1f40e2d10 2024-03-29T11:46:23.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.389+0000 7fe1e7fff640 10 client.4380.objecter ms_dispatch 0x7fe1fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:23.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.390+0000 7fe1e7fff640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:23.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.390+0000 7fe1e7fff640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:23.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.390+0000 7fe1e7fff640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T11:46:23.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.390+0000 7fe1fb7fe640 10 monclient: get_auth_request con 0x7fe1e006dd70 auth_method 0 2024-03-29T11:46:23.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.391+0000 7fe200d64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:23.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.391+0000 7fe1e7fff640 10 client.4380.objecter ms_handle_connect 0x7fe1e006dd70 2024-03-29T11:46:23.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.392+0000 7fe200d64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:23.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.392+0000 7fe200d64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:23.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.392+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc005080 con 0x7fe1f40e2d10 2024-03-29T11:46:23.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.401+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1f0050440 con 0x7fe1f40e2d10 2024-03-29T11:46:23.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.401+0000 7fe1e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:23.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.401+0000 7fe1e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:23.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.736+0000 7fe200d64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:23.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.736+0000 7fe200d64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:23.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.736+0000 7fe200d64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:23.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.736+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1dc005330 con 0x7fe1f40e2d10 2024-03-29T11:46:23.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.737+0000 7fe1e7fff640 1 -- 172.21.15.55:0/2591366103 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fe1f0052370 con 0x7fe1f40e2d10 2024-03-29T11:46:23.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.738+0000 7fe1e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:23.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.738+0000 7fe1e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:23.744 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:46:23.744 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:23.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.742+0000 7fe200d64640 10 librados: watch_flush enter 2024-03-29T11:46:23.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.742+0000 7fe200d64640 10 librados: watch_flush exit 2024-03-29T11:46:23.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.742+0000 7fe200d64640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T11:46:23.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.742+0000 7fe200d64640 10 client.4380.objecter successfully canceled tick 2024-03-29T11:46:23.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.742+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 >> v1:172.21.15.171:6816/1463197163 conn(0x7fe1e006dd70 legacy=0x7fe1e0070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:23.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.742+0000 7fe200d64640 10 monclient: shutdown 2024-03-29T11:46:23.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.742+0000 7fe200d64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:23.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.742+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 >> v1:172.21.15.55:6789/0 conn(0x7fe1f40e2d10 legacy=0x7fe1f40e3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:23.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.743+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 shutdown_connections 2024-03-29T11:46:23.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.743+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 >> 172.21.15.55:0/2591366103 conn(0x7fe1f40058f0 msgr2=0x7fe1f40299f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:23.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.743+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 shutdown_connections 2024-03-29T11:46:23.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.743+0000 7fe200d64640 1 -- 172.21.15.55:0/2591366103 wait complete. 2024-03-29T11:46:23.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:23.743+0000 7fe200d64640 1 librados: shutdown 2024-03-29T11:46:23.765 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:30.571 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:30.740 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.734+0000 7f4f45f3a640 10 monclient: get_monmap_and_config 2024-03-29T11:46:30.740 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.734+0000 7f4f45f3a640 10 monclient: build_initial_monmap 2024-03-29T11:46:30.740 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.734+0000 7f4f45f3a640 10 monclient: monmap: 2024-03-29T11:46:30.740 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:30.740 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:30.740 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:46:30.735290+0000 2024-03-29T11:46:30.740 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:46:30.735290+0000 2024-03-29T11:46:30.741 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:30.741 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:30.741 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:30.741 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:46:30.741 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:46:30.741 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.735+0000 7f4f45f3a640 1 Processor -- start 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.735+0000 7f4f45f3a640 1 -- start start 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.735+0000 7f4f45f3a640 10 monclient: init 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f4f400a9930 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f4f40149dc0 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f4f4014ff40 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f400aae60 con 0x7f4f40149dc0 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f400b4310 con 0x7f4f400a9930 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f400afdd0 con 0x7f4f4014ff40 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:30.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.736+0000 7f4f45f3a640 10 monclient(hunting): authenticate will time out at 1204.016442s 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.740+0000 7f4f3ffff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f4f4014ff40 0x7f4f40152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:60999/0 (socket says 172.21.15.55:60999) 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.740+0000 7f4f3ffff640 1 -- 172.21.15.55:0/2026277220 learned_addr learned my addr 172.21.15.55:0/2026277220 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.741+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711440360 0 0) 0x7f4f400aae60 con 0x7f4f40149dc0 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.741+0000 7f4f3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.741+0000 7f4f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.741+0000 7f4f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.741+0000 7f4f3e7fc640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.741+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f24003520 con 0x7f4f40149dc0 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.742+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643635094 0 0) 0x7f4f400afdd0 con 0x7f4f4014ff40 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.742+0000 7f4f3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.742+0000 7f4f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.742+0000 7f4f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.742+0000 7f4f3e7fc640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:46:30.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.742+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f400aae60 con 0x7f4f4014ff40 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916307227 0 0) 0x7f4f400aae60 con 0x7f4f4014ff40 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 >> v1:172.21.15.171:6789/0 conn(0x7f4f400a9930 legacy=0x7f4f40143a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 >> v1:172.21.15.55:6789/0 conn(0x7f4f40149dc0 legacy=0x7f4f4014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4014a950 con 0x7f4f4014ff40 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:46:30.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f45f3a640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f45f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4f380019f0 con 0x7f4f4014ff40 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 10 monclient: dump: 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:30.754 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f45f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 1 -- 172.21.15.55:0/2026277220 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f38001c10 con 0x7f4f4014ff40 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.743+0000 7f4f3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.744+0000 7f4f45f3a640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.744+0000 7f4f45f3a640 4 set_mon_vals no callback set 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.744+0000 7f4f45f3a640 10 monclient: shutdown 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.744+0000 7f4f45f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:30.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.744+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2026277220 >> v1:172.21.15.55:6790/0 conn(0x7f4f4014ff40 legacy=0x7f4f40152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.745+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2026277220 shutdown_connections 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.745+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2026277220 >> 172.21.15.55:0/2026277220 conn(0x7f4f400b35e0 msgr2=0x7f4f400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.747+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2026277220 shutdown_connections 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.747+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2026277220 wait complete. 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.748+0000 7f4f45f3a640 10 monclient: build_initial_monmap 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.748+0000 7f4f45f3a640 10 monclient: monmap: 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:46:30.756 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:30.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.749+0000 7f4f45f3a640 1 librados: starting msgr at 2024-03-29T11:46:30.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.749+0000 7f4f45f3a640 1 librados: starting objecter 2024-03-29T11:46:30.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.749+0000 7f4f45f3a640 1 Processor -- start 2024-03-29T11:46:30.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.749+0000 7f4f45f3a640 1 -- start start 2024-03-29T11:46:30.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.749+0000 7f4f45f3a640 1 librados: setting wanted keys 2024-03-29T11:46:30.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.749+0000 7f4f45f3a640 1 librados: calling monclient init 2024-03-29T11:46:30.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.749+0000 7f4f45f3a640 10 monclient: init 2024-03-29T11:46:30.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:30.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:46:30.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f4f400a9930 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:30.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f4f40149dc0 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:30.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f4f4014ff40 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:30.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f401db620 con 0x7f4f4014ff40 2024-03-29T11:46:30.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f400aca60 con 0x7f4f40149dc0 2024-03-29T11:46:30.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f401dc1c0 con 0x7f4f400a9930 2024-03-29T11:46:30.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:30.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.750+0000 7f4f45f3a640 10 monclient(hunting): authenticate will time out at 1204.030187s 2024-03-29T11:46:30.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.753+0000 7f4f3ffff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f4f4014ff40 0x7f4f401e5e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:50770/0 (socket says 172.21.15.55:50770) 2024-03-29T11:46:30.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.753+0000 7f4f3ffff640 1 -- 172.21.15.55:0/2878651146 learned_addr learned my addr 172.21.15.55:0/2878651146 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:30.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 client.?.objecter ms_handle_connect 0x7f4f4014ff40 2024-03-29T11:46:30.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:30.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 client.?.objecter ms_handle_connect 0x7f4f40149dc0 2024-03-29T11:46:30.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:30.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211311032 0 0) 0x7f4f401db620 con 0x7f4f4014ff40 2024-03-29T11:46:30.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:46:30.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:30.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:30.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:46:30.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f28004eb0 con 0x7f4f4014ff40 2024-03-29T11:46:30.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582613763 0 0) 0x7f4f400aca60 con 0x7f4f40149dc0 2024-03-29T11:46:30.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:46:30.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:30.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:30.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:46:30.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.754+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f401db620 con 0x7f4f40149dc0 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.757+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1730757203 0 0) 0x7f4f401db620 con 0x7f4f40149dc0 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.757+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f400aca60 con 0x7f4f40149dc0 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.757+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4f2c001950 con 0x7f4f40149dc0 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.757+0000 7f4f37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.757+0000 7f4f37fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.757+0000 7f4f37fff640 10 monclient(hunting): dump: 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:30.763 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1115748179 0 0) 0x7f4f28004eb0 con 0x7f4f4014ff40 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f401db620 con 0x7f4f4014ff40 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4f38004d00 con 0x7f4f4014ff40 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient(hunting): dump: 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:30.764 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:30.765 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:30.765 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:30.765 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:30.765 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:30.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 client.?.objecter ms_handle_connect 0x7f4f400a9930 2024-03-29T11:46:30.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:30.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3136334543 0 0) 0x7f4f401db620 con 0x7f4f4014ff40 2024-03-29T11:46:30.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 >> v1:172.21.15.55:6790/0 conn(0x7f4f400a9930 legacy=0x7f4f400ac350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:30.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 >> v1:172.21.15.171:6789/0 conn(0x7f4f40149dc0 legacy=0x7f4f401dbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:30.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient: _finish_hunting 0 2024-03-29T11:46:30.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 monclient: found mon.a 2024-03-29T11:46:30.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:30.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:30.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f400ab240 con 0x7f4f4014ff40 2024-03-29T11:46:30.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient: _finish_auth 0 2024-03-29T11:46:30.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient: _check_auth_tickets 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f45f3a640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f45f3a640 10 monclient: _renew_subs 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f45f3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f400aba70 con 0x7f4f4014ff40 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f45f3a640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f45f3a640 10 monclient: _renew_subs 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f45f3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f40147a10 con 0x7f4f4014ff40 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f45f3a640 1 librados: init done 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f38005150 con 0x7f4f4014ff40 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:30.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4f38005560 con 0x7f4f4014ff40 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f37fff640 10 monclient: dump: 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:30.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.758+0000 7f4f3dffb640 4 set_mon_vals no callback set 2024-03-29T11:46:30.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.761+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f4f38001d80 con 0x7f4f4014ff40 2024-03-29T11:46:30.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f4f38084f60 con 0x7f4f4014ff40 2024-03-29T11:46:30.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f37fff640 10 client.4386.objecter ms_dispatch 0x7f4f40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:30.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f37fff640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:30.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f37fff640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:30.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f37fff640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-29T11:46:30.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f45f3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:30.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f45f3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:30.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f45f3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:30.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c005080 con 0x7f4f4014ff40 2024-03-29T11:46:30.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.762+0000 7f4f3f7fe640 10 monclient: get_auth_request con 0x7f4f2806df50 auth_method 0 2024-03-29T11:46:30.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.763+0000 7f4f37fff640 10 client.4386.objecter ms_handle_connect 0x7f4f2806df50 2024-03-29T11:46:30.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.773+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f38050840 con 0x7f4f4014ff40 2024-03-29T11:46:30.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.773+0000 7f4f37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:30.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:30.773+0000 7f4f37fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:31.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.176+0000 7f4f45f3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:31.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.176+0000 7f4f45f3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:31.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.176+0000 7f4f45f3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:31.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.177+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f1c005330 con 0x7f4f4014ff40 2024-03-29T11:46:31.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.178+0000 7f4f37fff640 1 -- 172.21.15.55:0/2878651146 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f4f38052770 con 0x7f4f4014ff40 2024-03-29T11:46:31.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.178+0000 7f4f37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:31.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.178+0000 7f4f37fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:31.184 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:46:31.184 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:31.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.182+0000 7f4f45f3a640 10 librados: watch_flush enter 2024-03-29T11:46:31.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.184+0000 7f4f45f3a640 10 librados: watch_flush exit 2024-03-29T11:46:31.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.184+0000 7f4f45f3a640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-29T11:46:31.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.184+0000 7f4f45f3a640 10 client.4386.objecter successfully canceled tick 2024-03-29T11:46:31.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.185+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 >> v1:172.21.15.171:6816/1463197163 conn(0x7f4f2806df50 legacy=0x7f4f28070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:31.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.185+0000 7f4f45f3a640 10 monclient: shutdown 2024-03-29T11:46:31.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.185+0000 7f4f45f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:31.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.185+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 >> v1:172.21.15.55:6789/0 conn(0x7f4f4014ff40 legacy=0x7f4f401e5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:31.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.185+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 shutdown_connections 2024-03-29T11:46:31.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.185+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 >> 172.21.15.55:0/2878651146 conn(0x7f4f400b35e0 msgr2=0x7f4f401408b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:31.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.186+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 shutdown_connections 2024-03-29T11:46:31.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.186+0000 7f4f45f3a640 1 -- 172.21.15.55:0/2878651146 wait complete. 2024-03-29T11:46:31.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:31.186+0000 7f4f45f3a640 1 librados: shutdown 2024-03-29T11:46:31.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:38.018 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:38.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.163+0000 7f7db26a8640 10 monclient: get_monmap_and_config 2024-03-29T11:46:38.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.163+0000 7f7db26a8640 10 monclient: build_initial_monmap 2024-03-29T11:46:38.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.163+0000 7f7db26a8640 10 monclient: monmap: 2024-03-29T11:46:38.171 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:38.171 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:38.171 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:46:38.163728+0000 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:46:38.163728+0000 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.165+0000 7f7db26a8640 1 Processor -- start 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.165+0000 7f7db26a8640 1 -- start start 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.165+0000 7f7db26a8640 10 monclient: init 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 10 monclient(hunting): picked mon.noname-b con 0x7f7dac1457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:38.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 10 monclient(hunting): picked mon.noname-a con 0x7f7dac13f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 10 monclient(hunting): picked mon.noname-c con 0x7f7dac0a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac0aaef0 con 0x7f7dac13f650 2024-03-29T11:46:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac145bd0 con 0x7f7dac0a9d70 2024-03-29T11:46:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac145eb0 con 0x7f7dac1457f0 2024-03-29T11:46:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.166+0000 7f7db26a8640 10 monclient(hunting): authenticate will time out at 1211.445623s 2024-03-29T11:46:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.167+0000 7f7dabfff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f7dac1457f0 0x7f7dac13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:37714/0 (socket says 172.21.15.55:37714) 2024-03-29T11:46:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.167+0000 7f7dabfff640 1 -- 172.21.15.55:0/3995271765 learned_addr learned my addr 172.21.15.55:0/3995271765 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.167+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415406590 0 0) 0x7f7dac145bd0 con 0x7f7dac0a9d70 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.167+0000 7f7daaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.167+0000 7f7daaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.167+0000 7f7daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.167+0000 7f7daaffd640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.167+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d98003520 con 0x7f7dac0a9d70 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913215194 0 0) 0x7f7dac145eb0 con 0x7f7dac1457f0 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:46:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac145bd0 con 0x7f7dac1457f0 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184654750 0 0) 0x7f7dac0aaef0 con 0x7f7dac13f650 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac145eb0 con 0x7f7dac13f650 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369300371 0 0) 0x7f7dac145bd0 con 0x7f7dac1457f0 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 >> v1:172.21.15.171:6789/0 conn(0x7f7dac0a9d70 legacy=0x7f7dac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 >> v1:172.21.15.55:6789/0 conn(0x7f7dac13f650 legacy=0x7f7dac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 monclient: found mon.noname-b 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac146340 con 0x7f7dac1457f0 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient: _finish_auth 0 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7d94001670 con 0x7f7dac1457f0 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7daaffd640 10 monclient: dump: 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.178 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7db26a8640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.168+0000 7f7db26a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.169+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d94001850 con 0x7f7dac1457f0 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.169+0000 7f7daaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.169+0000 7f7daaffd640 1 -- 172.21.15.55:0/3995271765 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7d94001c60 con 0x7f7dac1457f0 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.169+0000 7f7daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.169+0000 7f7daaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.169+0000 7f7daaffd640 10 monclient: dump: 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:38.179 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.169+0000 7f7db26a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.169+0000 7f7db26a8640 4 set_mon_vals no callback set 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.170+0000 7f7db26a8640 10 monclient: shutdown 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.170+0000 7f7db26a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.170+0000 7f7db26a8640 1 -- 172.21.15.55:0/3995271765 >> v1:172.21.15.55:6790/0 conn(0x7f7dac1457f0 legacy=0x7f7dac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.170+0000 7f7db26a8640 1 -- 172.21.15.55:0/3995271765 shutdown_connections 2024-03-29T11:46:38.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.170+0000 7f7db26a8640 1 -- 172.21.15.55:0/3995271765 >> 172.21.15.55:0/3995271765 conn(0x7f7dac066a00 msgr2=0x7f7dac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.170+0000 7f7db26a8640 1 -- 172.21.15.55:0/3995271765 shutdown_connections 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.170+0000 7f7db26a8640 1 -- 172.21.15.55:0/3995271765 wait complete. 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.171+0000 7f7db26a8640 10 monclient: build_initial_monmap 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.172+0000 7f7db26a8640 10 monclient: monmap: 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:46:38.181 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.172+0000 7f7db26a8640 1 librados: starting msgr at 2024-03-29T11:46:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.172+0000 7f7db26a8640 1 librados: starting objecter 2024-03-29T11:46:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.172+0000 7f7db26a8640 1 Processor -- start 2024-03-29T11:46:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.173+0000 7f7db26a8640 1 -- start start 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.173+0000 7f7db26a8640 1 librados: setting wanted keys 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.173+0000 7f7db26a8640 1 librados: calling monclient init 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.173+0000 7f7db26a8640 10 monclient: init 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7db26a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7db26a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7db26a8640 10 monclient(hunting): picked mon.noname-c con 0x7f7dac0a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7db26a8640 10 monclient(hunting): picked mon.noname-b con 0x7f7dac13f650 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7db26a8640 10 monclient(hunting): picked mon.noname-a con 0x7f7dac1457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7dab7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7dac13f650 0x7f7dac0b0d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:36492/0 (socket says 172.21.15.55:36492) 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7dab7fe640 1 -- 172.21.15.55:0/3636743922 learned_addr learned my addr 172.21.15.55:0/3636743922 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac1427f0 con 0x7f7dac1457f0 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7da8ff9640 10 client.?.objecter ms_handle_connect 0x7f7dac0a9d70 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7da8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:38.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.174+0000 7f7da8ff9640 10 client.?.objecter ms_handle_connect 0x7f7dac13f650 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7da8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac137a10 con 0x7f7dac13f650 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7da8ff9640 10 client.?.objecter ms_handle_connect 0x7f7dac1457f0 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7da8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac142c80 con 0x7f7dac0a9d70 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939367850 0 0) 0x7f7dac1427f0 con 0x7f7dac1457f0 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7db26a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7db26a8640 10 monclient(hunting): authenticate will time out at 1211.454402s 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7da8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7da8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7da8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.175+0000 7f7da8ff9640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d8c004eb0 con 0x7f7dac1457f0 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635228614 0 0) 0x7f7dac142c80 con 0x7f7dac0a9d70 2024-03-29T11:46:38.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac1427f0 con 0x7f7dac0a9d70 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496036163 0 0) 0x7f7dac137a10 con 0x7f7dac13f650 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.176+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac142c80 con 0x7f7dac13f650 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1753338909 0 0) 0x7f7dac1427f0 con 0x7f7dac0a9d70 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dac137a10 con 0x7f7dac0a9d70 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7d94001ba0 con 0x7f7dac0a9d70 2024-03-29T11:46:38.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): dump: 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:38.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 207349559 0 0) 0x7f7dac142c80 con 0x7f7dac13f650 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dac1427f0 con 0x7f7dac13f650 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7da4001bf0 con 0x7f7dac13f650 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): dump: 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:38.195 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3453135687 0 0) 0x7f7d8c004eb0 con 0x7f7dac1457f0 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dac142c80 con 0x7f7dac1457f0 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7d9c001950 con 0x7f7dac1457f0 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient(hunting): dump: 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:38.196 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2637812994 0 0) 0x7f7dac137a10 con 0x7f7dac0a9d70 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 >> v1:172.21.15.171:6789/0 conn(0x7f7dac13f650 legacy=0x7f7dac0b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 >> v1:172.21.15.55:6789/0 conn(0x7f7dac1457f0 legacy=0x7f7dac1e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 monclient: found mon.c 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac1422d0 con 0x7f7dac0a9d70 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:46:38.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7da8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7db26a8640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7db26a8640 10 monclient: _renew_subs 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7db26a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.177+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dac142520 con 0x7f7dac0a9d70 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7db26a8640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7db26a8640 10 monclient: _renew_subs 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7db26a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dac1dae00 con 0x7f7dac0a9d70 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7db26a8640 1 librados: init done 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d94004a90 con 0x7f7dac0a9d70 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7da8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7d94001230 con 0x7f7dac0a9d70 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7da8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:38.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7da8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.178+0000 7f7da8ff9640 10 monclient: dump: 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.179+0000 7f7daa7fc640 4 set_mon_vals no callback set 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.180+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f7d9401aa70 con 0x7f7dac0a9d70 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.181+0000 7f7db26a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:38.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.181+0000 7f7db26a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.181+0000 7f7db26a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.181+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d90005080 con 0x7f7dac0a9d70 2024-03-29T11:46:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.181+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f7d94084920 con 0x7f7dac0a9d70 2024-03-29T11:46:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.182+0000 7f7dab7fe640 10 monclient: get_auth_request con 0x7f7d8c076930 auth_method 0 2024-03-29T11:46:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.182+0000 7f7da8ff9640 10 client.4403.objecter ms_dispatch 0x7f7dac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.182+0000 7f7da8ff9640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.182+0000 7f7da8ff9640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.182+0000 7f7da8ff9640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T11:46:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.182+0000 7f7da8ff9640 10 client.4403.objecter ms_handle_connect 0x7f7d8c076930 2024-03-29T11:46:38.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.196+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d94050200 con 0x7f7dac0a9d70 2024-03-29T11:46:38.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.196+0000 7f7da8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:38.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.196+0000 7f7da8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:38.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.633+0000 7f7db26a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:38.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.633+0000 7f7db26a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:38.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.633+0000 7f7db26a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:38.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.633+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d90005330 con 0x7f7dac0a9d70 2024-03-29T11:46:38.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.634+0000 7f7da8ff9640 1 -- 172.21.15.55:0/3636743922 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f7d94052130 con 0x7f7dac0a9d70 2024-03-29T11:46:38.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.634+0000 7f7da8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:38.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.634+0000 7f7da8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:38.639 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:46:38.639 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:38.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.638+0000 7f7db26a8640 10 librados: watch_flush enter 2024-03-29T11:46:38.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.638+0000 7f7db26a8640 10 librados: watch_flush exit 2024-03-29T11:46:38.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.638+0000 7f7db26a8640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T11:46:38.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.638+0000 7f7db26a8640 10 client.4403.objecter successfully canceled tick 2024-03-29T11:46:38.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.639+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 >> v1:172.21.15.171:6816/1463197163 conn(0x7f7d8c076930 legacy=0x7f7d8c078df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:38.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.639+0000 7f7db26a8640 10 monclient: shutdown 2024-03-29T11:46:38.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.639+0000 7f7db26a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:38.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.639+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 >> v1:172.21.15.55:6790/0 conn(0x7f7dac0a9d70 legacy=0x7f7dac1da510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:38.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.639+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 shutdown_connections 2024-03-29T11:46:38.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.639+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 >> 172.21.15.55:0/3636743922 conn(0x7f7dac066a00 msgr2=0x7f7dac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:38.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.639+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 shutdown_connections 2024-03-29T11:46:38.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.639+0000 7f7db26a8640 1 -- 172.21.15.55:0/3636743922 wait complete. 2024-03-29T11:46:38.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:38.640+0000 7f7db26a8640 1 librados: shutdown 2024-03-29T11:46:38.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:45.467 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:45.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.606+0000 7f3bb416b640 10 monclient: get_monmap_and_config 2024-03-29T11:46:45.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.606+0000 7f3bb416b640 10 monclient: build_initial_monmap 2024-03-29T11:46:45.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.606+0000 7f3bb416b640 10 monclient: monmap: 2024-03-29T11:46:45.612 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:45.612 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:45.612 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:46:45.607968+0000 2024-03-29T11:46:45.612 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:46:45.607968+0000 2024-03-29T11:46:45.612 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:45.612 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:45.613 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:45.613 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:46:45.613 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:46:45.613 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.607+0000 7f3bb416b640 1 Processor -- start 2024-03-29T11:46:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.607+0000 7f3bb416b640 1 -- start start 2024-03-29T11:46:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.607+0000 7f3bb416b640 10 monclient: init 2024-03-29T11:46:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.608+0000 7f3bb416b640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.608+0000 7f3bb416b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:46:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.608+0000 7f3bb416b640 10 monclient(hunting): picked mon.noname-b con 0x7f3bac0b0770 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.609+0000 7f3bb416b640 10 monclient(hunting): picked mon.noname-c con 0x7f3bac14a700 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.609+0000 7f3bb416b640 10 monclient(hunting): picked mon.noname-a con 0x7f3bac0ac5a0 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:45.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.609+0000 7f3bb416b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac0aba70 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.609+0000 7f3bb416b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac0aecb0 con 0x7f3bac14a700 2024-03-29T11:46:45.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.609+0000 7f3bb416b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac0af150 con 0x7f3bac0b0770 2024-03-29T11:46:45.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.609+0000 7f3bb416b640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:45.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.609+0000 7f3bb416b640 10 monclient(hunting): authenticate will time out at 1218.888728s 2024-03-29T11:46:45.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.609+0000 7f3bb2708640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3bac0ac5a0 0x7f3bac151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:60424/0 (socket says 172.21.15.55:60424) 2024-03-29T11:46:45.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.609+0000 7f3bb2708640 1 -- 172.21.15.55:0/3930421599 learned_addr learned my addr 172.21.15.55:0/3930421599 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226035891 0 0) 0x7f3bac0aba70 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 10 monclient(hunting): _negotiate 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b98003520 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209399800 0 0) 0x7f3bac0af150 con 0x7f3bac0b0770 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 10 monclient(hunting): _negotiate 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.610+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bac0aba70 con 0x7f3bac0b0770 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1382140681 0 0) 0x7f3b98003520 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 >> v1:172.21.15.55:6790/0 conn(0x7f3bac0b0770 legacy=0x7f3bac149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 >> v1:172.21.15.171:6789/0 conn(0x7f3bac14a700 legacy=0x7f3bac0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: _finish_hunting 0 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 1 monclient: found mon.noname-a 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac0ac9d0 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: _finish_auth 0 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: _check_auth_tickets 2024-03-29T11:46:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3ba4001690 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: dump: 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:45.620 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb416b640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb416b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ba40017f0 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 1 -- 172.21.15.55:0/3930421599 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3ba4001c40 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb0f05640 10 monclient: dump: 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:45.621 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb416b640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.611+0000 7f3bb416b640 4 set_mon_vals no callback set 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.612+0000 7f3bb416b640 10 monclient: shutdown 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.612+0000 7f3bb416b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:45.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.612+0000 7f3bb416b640 1 -- 172.21.15.55:0/3930421599 >> v1:172.21.15.55:6789/0 conn(0x7f3bac0ac5a0 legacy=0x7f3bac151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.612+0000 7f3bb416b640 1 -- 172.21.15.55:0/3930421599 shutdown_connections 2024-03-29T11:46:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.612+0000 7f3bb416b640 1 -- 172.21.15.55:0/3930421599 >> 172.21.15.55:0/3930421599 conn(0x7f3bac06b660 msgr2=0x7f3bac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.613+0000 7f3bb416b640 1 -- 172.21.15.55:0/3930421599 shutdown_connections 2024-03-29T11:46:45.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.613+0000 7f3bb416b640 1 -- 172.21.15.55:0/3930421599 wait complete. 2024-03-29T11:46:45.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.614+0000 7f3bb416b640 10 monclient: build_initial_monmap 2024-03-29T11:46:45.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.614+0000 7f3bb416b640 10 monclient: monmap: 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.614+0000 7f3bb416b640 1 librados: starting msgr at 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.614+0000 7f3bb416b640 1 librados: starting objecter 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.615+0000 7f3bb416b640 1 Processor -- start 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.615+0000 7f3bb416b640 1 -- start start 2024-03-29T11:46:45.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.615+0000 7f3bb416b640 1 librados: setting wanted keys 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.615+0000 7f3bb416b640 1 librados: calling monclient init 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.615+0000 7f3bb416b640 10 monclient: init 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.615+0000 7f3bb416b640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.615+0000 7f3bb416b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb416b640 10 monclient(hunting): picked mon.noname-c con 0x7f3bac0ac5a0 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb416b640 10 monclient(hunting): picked mon.noname-b con 0x7f3bac0b0770 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb416b640 10 monclient(hunting): picked mon.noname-a con 0x7f3bac14a700 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb416b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac126630 con 0x7f3bac14a700 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb416b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac12b340 con 0x7f3bac0b0770 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb416b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac128fc0 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb416b640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb416b640 10 monclient(hunting): authenticate will time out at 1218.895725s 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb2708640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3bac14a700 0x7f3bac1e7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:60906/0 (socket says 172.21.15.55:60906) 2024-03-29T11:46:45.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3bb2708640 1 -- 172.21.15.55:0/1352635765 learned_addr learned my addr 172.21.15.55:0/1352635765 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3ba27fc640 10 client.?.objecter ms_handle_connect 0x7f3bac14a700 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3ba27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3ba27fc640 10 client.?.objecter ms_handle_connect 0x7f3bac0b0770 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.616+0000 7f3ba27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267902925 0 0) 0x7f3bac126630 con 0x7f3bac14a700 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b90004eb0 con 0x7f3bac14a700 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 client.?.objecter ms_handle_connect 0x7f3bac0ac5a0 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462431220 0 0) 0x7f3bac12b340 con 0x7f3bac0b0770 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:45.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bac126630 con 0x7f3bac0b0770 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043326988 0 0) 0x7f3bac128fc0 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bac12b340 con 0x7f3bac0ac5a0 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1015867326 0 0) 0x7f3b90004eb0 con 0x7f3bac14a700 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bac128fc0 con 0x7f3bac14a700 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3ba4001b00 con 0x7f3bac14a700 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): dump: 2024-03-29T11:46:45.632 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.617+0000 7f3ba27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3056353910 0 0) 0x7f3bac128fc0 con 0x7f3bac14a700 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 >> v1:172.21.15.55:6790/0 conn(0x7f3bac0ac5a0 legacy=0x7f3bac147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 >> v1:172.21.15.171:6789/0 conn(0x7f3bac0b0770 legacy=0x7f3bac1dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:45.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 1 monclient: found mon.a 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac129f00 con 0x7f3bac14a700 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 10 monclient: _finish_auth 0 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3bb416b640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3bb416b640 10 monclient: _renew_subs 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3bb416b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bac12a3b0 con 0x7f3bac14a700 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3bb416b640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3bb416b640 10 monclient: _renew_subs 2024-03-29T11:46:45.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3bb416b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bac1488b0 con 0x7f3bac14a700 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3bb416b640 1 librados: init done 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ba4004bf0 con 0x7f3bac14a700 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3ba4001d80 con 0x7f3bac14a700 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba27fc640 10 monclient: dump: 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:45.635 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.618+0000 7f3ba3fff640 4 set_mon_vals no callback set 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.620+0000 7f3bb416b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.620+0000 7f3bb416b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.620+0000 7f3bb416b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.620+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f3ba401ad70 con 0x7f3bac14a700 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.620+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b94005080 con 0x7f3bac14a700 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.621+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f3ba4084b10 con 0x7f3bac14a700 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.621+0000 7f3ba27fc640 10 client.4398.objecter ms_dispatch 0x7f3bac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.621+0000 7f3ba27fc640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.621+0000 7f3ba27fc640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:45.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.621+0000 7f3ba27fc640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T11:46:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.621+0000 7f3bb1f07640 10 monclient: get_auth_request con 0x7f3b9006def0 auth_method 0 2024-03-29T11:46:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.622+0000 7f3ba27fc640 10 client.4398.objecter ms_handle_connect 0x7f3b9006def0 2024-03-29T11:46:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.629+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ba40503f0 con 0x7f3bac14a700 2024-03-29T11:46:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.629+0000 7f3ba27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.629+0000 7f3ba27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:46.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.996+0000 7f3bb416b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:46.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.996+0000 7f3bb416b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:46.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.996+0000 7f3bb416b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:46:46.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.996+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b94005330 con 0x7f3bac14a700 2024-03-29T11:46:46.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.997+0000 7f3ba27fc640 1 -- 172.21.15.55:0/1352635765 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f3ba4001210 con 0x7f3bac14a700 2024-03-29T11:46:46.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.997+0000 7f3ba27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:46.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:45.997+0000 7f3ba27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:46.004 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:46:46.004 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:46.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 10 librados: watch_flush enter 2024-03-29T11:46:46.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 10 librados: watch_flush exit 2024-03-29T11:46:46.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T11:46:46.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 10 client.4398.objecter successfully canceled tick 2024-03-29T11:46:46.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 >> v1:172.21.15.171:6816/1463197163 conn(0x7f3b9006def0 legacy=0x7f3b900703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:46.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 10 monclient: shutdown 2024-03-29T11:46:46.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:46.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 >> v1:172.21.15.55:6789/0 conn(0x7f3bac14a700 legacy=0x7f3bac1e7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:46.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 shutdown_connections 2024-03-29T11:46:46.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 >> 172.21.15.55:0/1352635765 conn(0x7f3bac06b660 msgr2=0x7f3bac149510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:46.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.001+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 shutdown_connections 2024-03-29T11:46:46.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.002+0000 7f3bb416b640 1 -- 172.21.15.55:0/1352635765 wait complete. 2024-03-29T11:46:46.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:46.002+0000 7f3bb416b640 1 librados: shutdown 2024-03-29T11:46:46.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:52.828 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:52.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.968+0000 7f60e7c9a640 10 monclient: get_monmap_and_config 2024-03-29T11:46:52.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.968+0000 7f60e7c9a640 10 monclient: build_initial_monmap 2024-03-29T11:46:52.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.968+0000 7f60e7c9a640 10 monclient: monmap: 2024-03-29T11:46:52.976 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:52.976 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:52.976 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:46:52.969622+0000 2024-03-29T11:46:52.976 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:46:52.969622+0000 2024-03-29T11:46:52.976 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:52.976 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.969+0000 7f60e7c9a640 1 Processor -- start 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.969+0000 7f60e7c9a640 1 -- start start 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.969+0000 7f60e7c9a640 10 monclient: init 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f60e00ad6c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f60e00af6c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f60e014fdc0 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e00a94e0 con 0x7f60e00af6c0 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e00af3a0 con 0x7f60e00ad6c0 2024-03-29T11:46:52.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e00afb60 con 0x7f60e014fdc0 2024-03-29T11:46:52.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:52.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.970+0000 7f60e7c9a640 10 monclient(hunting): authenticate will time out at 1226.250154s 2024-03-29T11:46:52.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.975+0000 7f60e5a36640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f60e00ad6c0 0x7f60e014f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:34885/0 (socket says 172.21.15.55:34885) 2024-03-29T11:46:52.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.975+0000 7f60e5a36640 1 -- 172.21.15.55:0/10445405 learned_addr learned my addr 172.21.15.55:0/10445405 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 1 -- 172.21.15.55:0/10445405 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638092881 0 0) 0x7f60e00a94e0 con 0x7f60e00af6c0 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 10 monclient(hunting): _negotiate 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 1 -- 172.21.15.55:0/10445405 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60cc003520 con 0x7f60e00af6c0 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 1 -- 172.21.15.55:0/10445405 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 929780513 0 0) 0x7f60cc003520 con 0x7f60e00af6c0 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 1 -- 172.21.15.55:0/10445405 >> v1:172.21.15.55:6790/0 conn(0x7f60e014fdc0 legacy=0x7f60e015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 1 -- 172.21.15.55:0/10445405 >> v1:172.21.15.171:6789/0 conn(0x7f60e00ad6c0 legacy=0x7f60e014f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 10 monclient: _finish_hunting 0 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 1 monclient: found mon.noname-a 2024-03-29T11:46:52.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.976+0000 7f60e4a34640 1 -- 172.21.15.55:0/10445405 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e00aab80 con 0x7f60e00af6c0 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 10 monclient: _finish_auth 0 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 10 monclient: _check_auth_tickets 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e7c9a640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e7c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 1 -- 172.21.15.55:0/10445405 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f60dc001690 con 0x7f60e00af6c0 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 10 monclient: dump: 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:52.986 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e7c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 1 -- 172.21.15.55:0/10445405 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60dc001920 con 0x7f60e00af6c0 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 1 -- 172.21.15.55:0/10445405 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f60dc001cb0 con 0x7f60e00af6c0 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e7c9a640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e7c9a640 4 set_mon_vals no callback set 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:52.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.977+0000 7f60e4a34640 10 monclient: dump: 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.978+0000 7f60e7c9a640 10 monclient: shutdown 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.978+0000 7f60e7c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.978+0000 7f60e7c9a640 1 -- 172.21.15.55:0/10445405 >> v1:172.21.15.55:6789/0 conn(0x7f60e00af6c0 legacy=0x7f60e0152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.978+0000 7f60e7c9a640 1 -- 172.21.15.55:0/10445405 shutdown_connections 2024-03-29T11:46:52.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.978+0000 7f60e7c9a640 1 -- 172.21.15.55:0/10445405 >> 172.21.15.55:0/10445405 conn(0x7f60e00b35e0 msgr2=0x7f60e00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:52.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.978+0000 7f60e7c9a640 1 -- 172.21.15.55:0/10445405 shutdown_connections 2024-03-29T11:46:52.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.978+0000 7f60e7c9a640 1 -- 172.21.15.55:0/10445405 wait complete. 2024-03-29T11:46:52.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.979+0000 7f60e7c9a640 10 monclient: build_initial_monmap 2024-03-29T11:46:52.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.979+0000 7f60e7c9a640 10 monclient: monmap: 2024-03-29T11:46:52.991 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:46:52.991 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.979+0000 7f60e7c9a640 1 librados: starting msgr at 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.979+0000 7f60e7c9a640 1 librados: starting objecter 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.979+0000 7f60e7c9a640 1 Processor -- start 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.979+0000 7f60e7c9a640 1 -- start start 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.979+0000 7f60e7c9a640 1 librados: setting wanted keys 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.979+0000 7f60e7c9a640 1 librados: calling monclient init 2024-03-29T11:46:52.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.979+0000 7f60e7c9a640 10 monclient: init 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f60e00ad6c0 addr v1:172.21.15.55:6790/0 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f60e00af6c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f60e014fdc0 addr v1:172.21.15.171:6789/0 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e01f2c20 con 0x7f60e00af6c0 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e00722c0 con 0x7f60e014fdc0 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e0071f20 con 0x7f60e00ad6c0 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.980+0000 7f60e7c9a640 10 monclient(hunting): authenticate will time out at 1226.260395s 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.984+0000 7f60e5a36640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f60e00ad6c0 0x7f60e01f85a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:43531/0 (socket says 172.21.15.55:43531) 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.984+0000 7f60e5a36640 1 -- 172.21.15.55:0/2658217291 learned_addr learned my addr 172.21.15.55:0/2658217291 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.984+0000 7f60d67fc640 10 client.?.objecter ms_handle_connect 0x7f60e00ad6c0 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.984+0000 7f60d67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:52.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.984+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262341948 0 0) 0x7f60e0071f20 con 0x7f60e00ad6c0 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.984+0000 7f60d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.984+0000 7f60d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.984+0000 7f60d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.984+0000 7f60d67fc640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c4004eb0 con 0x7f60e00ad6c0 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2730258526 0 0) 0x7f60c4004eb0 con 0x7f60e00ad6c0 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60e0071f20 con 0x7f60e00ad6c0 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f60d0001510 con 0x7f60e00ad6c0 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 10 monclient(hunting): dump: 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:52.994 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 831024568 0 0) 0x7f60e0071f20 con 0x7f60e00ad6c0 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 >> v1:172.21.15.171:6789/0 conn(0x7f60e014fdc0 legacy=0x7f60e01f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:52.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.985+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 >> v1:172.21.15.55:6789/0 conn(0x7f60e00af6c0 legacy=0x7f60e01f1e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 1 monclient: found mon.c 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e007aff0 con 0x7f60e00ad6c0 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 10 monclient: _finish_auth 0 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60e7c9a640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60e7c9a640 10 monclient: _renew_subs 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60e7c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:52.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e0071480 con 0x7f60e00ad6c0 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60e7c9a640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60e7c9a640 10 monclient: _renew_subs 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60e7c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e014e4c0 con 0x7f60e00ad6c0 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60e7c9a640 1 librados: init done 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60d0001ac0 con 0x7f60e00ad6c0 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f60d0004c30 con 0x7f60e00ad6c0 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d67fc640 10 monclient: dump: 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:53.000 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.987+0000 7f60d7fff640 4 set_mon_vals no callback set 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.989+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f60d001ac00 con 0x7f60e00ad6c0 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.990+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f60d0084b60 con 0x7f60e00ad6c0 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.990+0000 7f60d67fc640 10 client.4412.objecter ms_dispatch 0x7f60e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.990+0000 7f60d67fc640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.990+0000 7f60d67fc640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.990+0000 7f60d67fc640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.990+0000 7f60e5235640 10 monclient: get_auth_request con 0x7f60c406ddf0 auth_method 0 2024-03-29T11:46:53.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.990+0000 7f60d67fc640 10 client.4412.objecter ms_handle_connect 0x7f60c406ddf0 2024-03-29T11:46:53.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.992+0000 7f60e7c9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:53.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.992+0000 7f60e7c9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:53.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.992+0000 7f60e7c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:53.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:52.992+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c8005080 con 0x7f60e00ad6c0 2024-03-29T11:46:53.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.005+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60d0050440 con 0x7f60e00ad6c0 2024-03-29T11:46:53.011 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.005+0000 7f60d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:53.011 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.005+0000 7f60d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:53.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.334+0000 7f60e7c9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:53.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.334+0000 7f60e7c9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:53.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.334+0000 7f60e7c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:46:53.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.334+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60c8005330 con 0x7f60e00ad6c0 2024-03-29T11:46:53.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.336+0000 7f60d67fc640 1 -- 172.21.15.55:0/2658217291 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f60d0052370 con 0x7f60e00ad6c0 2024-03-29T11:46:53.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.336+0000 7f60d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:53.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.336+0000 7f60d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:53.341 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:46:53.341 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:53.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.339+0000 7f60e7c9a640 10 librados: watch_flush enter 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.339+0000 7f60e7c9a640 10 librados: watch_flush exit 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.339+0000 7f60e7c9a640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.339+0000 7f60e7c9a640 10 client.4412.objecter successfully canceled tick 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.339+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 >> v1:172.21.15.171:6816/1463197163 conn(0x7f60c406ddf0 legacy=0x7f60c40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.339+0000 7f60e7c9a640 10 monclient: shutdown 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.339+0000 7f60e7c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.339+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 >> v1:172.21.15.55:6790/0 conn(0x7f60e00ad6c0 legacy=0x7f60e01f85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.340+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 shutdown_connections 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.340+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 >> 172.21.15.55:0/2658217291 conn(0x7f60e00b35e0 msgr2=0x7f60e00a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.340+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 shutdown_connections 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.340+0000 7f60e7c9a640 1 -- 172.21.15.55:0/2658217291 wait complete. 2024-03-29T11:46:53.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:46:53.340+0000 7f60e7c9a640 1 librados: shutdown 2024-03-29T11:46:53.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:00.174 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:00.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.331+0000 7f3510f67640 10 monclient: get_monmap_and_config 2024-03-29T11:47:00.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.331+0000 7f3510f67640 10 monclient: build_initial_monmap 2024-03-29T11:47:00.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.331+0000 7f3510f67640 10 monclient: monmap: 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:47:00.332601+0000 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:47:00.332601+0000 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:47:00.338 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:00.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.334+0000 7f3510f67640 1 Processor -- start 2024-03-29T11:47:00.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.334+0000 7f3510f67640 1 -- start start 2024-03-29T11:47:00.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.334+0000 7f3510f67640 10 monclient: init 2024-03-29T11:47:00.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.335+0000 7f3510f67640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:00.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.335+0000 7f3510f67640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:47:00.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.335+0000 7f3510f67640 10 monclient(hunting): picked mon.noname-b con 0x7f350c0a9750 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:00.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.335+0000 7f3510f67640 10 monclient(hunting): picked mon.noname-a con 0x7f350c149890 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:00.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.335+0000 7f3510f67640 10 monclient(hunting): picked mon.noname-c con 0x7f350c0afa30 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:00.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.336+0000 7f3510f67640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c0b00c0 con 0x7f350c149890 2024-03-29T11:47:00.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.336+0000 7f3510f67640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c1464a0 con 0x7f350c0afa30 2024-03-29T11:47:00.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.336+0000 7f3510f67640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c0ab280 con 0x7f350c0a9750 2024-03-29T11:47:00.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.336+0000 7f3510f67640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:00.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.336+0000 7f3510f67640 10 monclient(hunting): authenticate will time out at 1233.615671s 2024-03-29T11:47:00.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.339+0000 7f350a59c640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f350c0a9750 0x7f350c1469a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:59451/0 (socket says 172.21.15.55:59451) 2024-03-29T11:47:00.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.339+0000 7f350a59c640 1 -- 172.21.15.55:0/1924948196 learned_addr learned my addr 172.21.15.55:0/1924948196 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137573588 0 0) 0x7f350c1464a0 con 0x7f350c0afa30 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 10 monclient(hunting): _negotiate 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f4003520 con 0x7f350c0afa30 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968179041 0 0) 0x7f350c0ab280 con 0x7f350c0a9750 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 10 monclient(hunting): _negotiate 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:47:00.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.341+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350c1464a0 con 0x7f350c0a9750 2024-03-29T11:47:00.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3284517373 0 0) 0x7f34f4003520 con 0x7f350c0afa30 2024-03-29T11:47:00.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 >> v1:172.21.15.55:6790/0 conn(0x7f350c0a9750 legacy=0x7f350c1469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:00.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 >> v1:172.21.15.55:6789/0 conn(0x7f350c149890 legacy=0x7f350c149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:00.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: _finish_hunting 0 2024-03-29T11:47:00.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 1 monclient: found mon.noname-c 2024-03-29T11:47:00.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:00.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c1470b0 con 0x7f350c0afa30 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: _finish_auth 0 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: _check_auth_tickets 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f3510f67640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f3510f67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3504001940 con 0x7f350c0afa30 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:47:00.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: dump: 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:00.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f3510f67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3504001b60 con 0x7f350c0afa30 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f3510f67640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f3510f67640 4 set_mon_vals no callback set 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 1 -- 172.21.15.55:0/1924948196 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3504001ef0 con 0x7f350c0afa30 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.342+0000 7f350959a640 10 monclient: dump: 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:00.353 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.343+0000 7f3510f67640 10 monclient: shutdown 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.343+0000 7f3510f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:00.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.343+0000 7f3510f67640 1 -- 172.21.15.55:0/1924948196 >> v1:172.21.15.171:6789/0 conn(0x7f350c0afa30 legacy=0x7f350c151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.345+0000 7f3510f67640 1 -- 172.21.15.55:0/1924948196 shutdown_connections 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.345+0000 7f3510f67640 1 -- 172.21.15.55:0/1924948196 >> 172.21.15.55:0/1924948196 conn(0x7f350c0b3a00 msgr2=0x7f350c0b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.346+0000 7f3510f67640 1 -- 172.21.15.55:0/1924948196 shutdown_connections 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.349+0000 7f3510f67640 1 -- 172.21.15.55:0/1924948196 wait complete. 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.350+0000 7f3510f67640 10 monclient: build_initial_monmap 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.350+0000 7f3510f67640 10 monclient: monmap: 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:47:00.356 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:47:00.357 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:00.357 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:00.357 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:00.357 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:47:00.357 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:47:00.357 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:00.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.350+0000 7f3510f67640 1 librados: starting msgr at 2024-03-29T11:47:00.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.350+0000 7f3510f67640 1 librados: starting objecter 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.351+0000 7f3510f67640 1 Processor -- start 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.351+0000 7f3510f67640 1 -- start start 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.351+0000 7f3510f67640 1 librados: setting wanted keys 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.351+0000 7f3510f67640 1 librados: calling monclient init 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.351+0000 7f3510f67640 10 monclient: init 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.351+0000 7f3510f67640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.351+0000 7f3510f67640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.351+0000 7f3510f67640 10 monclient(hunting): picked mon.noname-a con 0x7f350c0a9750 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.352+0000 7f3510f67640 10 monclient(hunting): picked mon.noname-c con 0x7f350c0afa30 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.352+0000 7f3510f67640 10 monclient(hunting): picked mon.noname-b con 0x7f350c149890 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.352+0000 7f3510f67640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c06e2e0 con 0x7f350c0a9750 2024-03-29T11:47:00.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.352+0000 7f3510f67640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c06e530 con 0x7f350c149890 2024-03-29T11:47:00.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.352+0000 7f3510f67640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c148d90 con 0x7f350c0afa30 2024-03-29T11:47:00.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.352+0000 7f3510f67640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:00.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.352+0000 7f3510f67640 10 monclient(hunting): authenticate will time out at 1233.631686s 2024-03-29T11:47:00.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f350a59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f350c0a9750 0x7f350c142ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:57803/0 (socket says 172.21.15.55:57803) 2024-03-29T11:47:00.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f350a59c640 1 -- 172.21.15.55:0/721806475 learned_addr learned my addr 172.21.15.55:0/721806475 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:00.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 client.?.objecter ms_handle_connect 0x7f350c0afa30 2024-03-29T11:47:00.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 client.?.objecter ms_handle_connect 0x7f350c0a9750 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383937833 0 0) 0x7f350c148d90 con 0x7f350c0afa30 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ec004eb0 con 0x7f350c0afa30 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543095210 0 0) 0x7f350c06e2e0 con 0x7f350c0a9750 2024-03-29T11:47:00.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.358+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350c148d90 con 0x7f350c0a9750 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.359+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1328514621 0 0) 0x7f350c148d90 con 0x7f350c0a9750 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.359+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f350c06e2e0 con 0x7f350c0a9750 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 client.?.objecter ms_handle_connect 0x7f350c149890 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f34fc001970 con 0x7f350c0a9750 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient(hunting): dump: 2024-03-29T11:47:00.373 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922717784 0 0) 0x7f350c06e530 con 0x7f350c149890 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:00.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350c148d90 con 0x7f350c149890 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2661193026 0 0) 0x7f34ec004eb0 con 0x7f350c0afa30 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f350c06e530 con 0x7f350c0afa30 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1087196461 0 0) 0x7f350c06e2e0 con 0x7f350c0a9750 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 >> v1:172.21.15.55:6790/0 conn(0x7f350c0afa30 legacy=0x7f350c141c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 >> v1:172.21.15.171:6789/0 conn(0x7f350c149890 legacy=0x7f350c1e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient: _finish_hunting 0 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 monclient: found mon.a 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c148f20 con 0x7f350c0a9750 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient: _finish_auth 0 2024-03-29T11:47:00.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 10 monclient: _check_auth_tickets 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f34faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f3510f67640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f3510f67640 10 monclient: _renew_subs 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f3510f67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.362+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c140100 con 0x7f350c0a9750 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f3510f67640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f3510f67640 10 monclient: _renew_subs 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f3510f67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c11b400 con 0x7f350c0a9750 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f3510f67640 1 librados: init done 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34fc0047e0 con 0x7f350c0a9750 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f34faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f34fc004af0 con 0x7f350c0a9750 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f34faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:00.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.363+0000 7f3508d99640 4 set_mon_vals no callback set 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.364+0000 7f34faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.364+0000 7f34faffd640 10 monclient: dump: 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:00.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.364+0000 7f3510f67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:00.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.365+0000 7f3510f67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:00.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.365+0000 7f3510f67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:00.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.365+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f0005080 con 0x7f350c0a9750 2024-03-29T11:47:00.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.367+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f34fc01aac0 con 0x7f350c0a9750 2024-03-29T11:47:00.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.368+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f34fc084a20 con 0x7f350c0a9750 2024-03-29T11:47:00.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.368+0000 7f34faffd640 10 client.4407.objecter ms_dispatch 0x7f350c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:47:00.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.368+0000 7f34faffd640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:47:00.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.368+0000 7f34faffd640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:47:00.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.368+0000 7f34faffd640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T11:47:00.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.369+0000 7f3509d9b640 10 monclient: get_auth_request con 0x7f34ec07edc0 auth_method 0 2024-03-29T11:47:00.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.369+0000 7f34faffd640 10 client.4407.objecter ms_handle_connect 0x7f34ec07edc0 2024-03-29T11:47:00.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.377+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34fc050300 con 0x7f350c0a9750 2024-03-29T11:47:00.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.377+0000 7f34faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:00.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.377+0000 7f34faffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:00.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.713+0000 7f3510f67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:00.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.713+0000 7f3510f67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:00.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.713+0000 7f3510f67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:00.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.713+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34f0005600 con 0x7f350c0a9750 2024-03-29T11:47:00.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.716+0000 7f34faffd640 1 -- 172.21.15.55:0/721806475 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f34fc052230 con 0x7f350c0a9750 2024-03-29T11:47:00.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.716+0000 7f34faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:00.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.716+0000 7f34faffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:00.722 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:47:00.722 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:00.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.719+0000 7f3510f67640 10 librados: watch_flush enter 2024-03-29T11:47:00.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.721+0000 7f3510f67640 10 librados: watch_flush exit 2024-03-29T11:47:00.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.721+0000 7f3510f67640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T11:47:00.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.721+0000 7f3510f67640 10 client.4407.objecter successfully canceled tick 2024-03-29T11:47:00.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.721+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 >> v1:172.21.15.171:6816/1463197163 conn(0x7f34ec07edc0 legacy=0x7f34ec081280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:00.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.721+0000 7f3510f67640 10 monclient: shutdown 2024-03-29T11:47:00.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.721+0000 7f3510f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:00.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.721+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 >> v1:172.21.15.55:6789/0 conn(0x7f350c0a9750 legacy=0x7f350c142ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:00.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.721+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 shutdown_connections 2024-03-29T11:47:00.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.722+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 >> 172.21.15.55:0/721806475 conn(0x7f350c0b3a00 msgr2=0x7f350c0ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:00.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.722+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 shutdown_connections 2024-03-29T11:47:00.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.722+0000 7f3510f67640 1 -- 172.21.15.55:0/721806475 wait complete. 2024-03-29T11:47:00.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:00.722+0000 7f3510f67640 1 librados: shutdown 2024-03-29T11:47:00.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:07.552 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:07.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.701+0000 7fa3d3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.701+0000 7fa3d3fff640 10 monclient: build_initial_monmap 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.701+0000 7fa3d3fff640 10 monclient: monmap: 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:47:07.702379+0000 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:47:07.702379+0000 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:47:07.707 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:07.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.702+0000 7fa3d3fff640 1 Processor -- start 2024-03-29T11:47:07.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.702+0000 7fa3d3fff640 1 -- start start 2024-03-29T11:47:07.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.702+0000 7fa3d3fff640 10 monclient: init 2024-03-29T11:47:07.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:07.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa3cc0066e0 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa3cc0cf820 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa3cc0c92b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc02c100 con 0x7fa3cc0cf820 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc02c3e0 con 0x7fa3cc0066e0 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc02a030 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.703+0000 7fa3d3fff640 10 monclient(hunting): authenticate will time out at 1240.982943s 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.706+0000 7fa3d37fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fa3cc0c92b0 0x7fa3cc0d74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:55493/0 (socket says 172.21.15.55:55493) 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.706+0000 7fa3d37fe640 1 -- 172.21.15.55:0/1092613574 learned_addr learned my addr 172.21.15.55:0/1092613574 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.707+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271500787 0 0) 0x7fa3cc02c100 con 0x7fa3cc0cf820 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.707+0000 7fa3d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:47:07.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.707+0000 7fa3d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.707+0000 7fa3d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.707+0000 7fa3d1ffb640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.707+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3bc003520 con 0x7fa3cc0cf820 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.708+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829838585 0 0) 0x7fa3cc02a030 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.708+0000 7fa3d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.708+0000 7fa3d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.708+0000 7fa3d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.708+0000 7fa3d1ffb640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.708+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3cc02c100 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.708+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3972703235 0 0) 0x7fa3cc02c100 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.708+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 >> v1:172.21.15.171:6789/0 conn(0x7fa3cc0066e0 legacy=0x7fa3cc0cf110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 >> v1:172.21.15.55:6789/0 conn(0x7fa3cc0cf820 legacy=0x7fa3cc0c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc029ce0 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa3c8001670 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.711+0000 7fa3d1ffb640 10 monclient: dump: 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.712+0000 7fa3d3fff640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:47:07.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.712+0000 7fa3d3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.715+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3c8001850 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.715+0000 7fa3d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.715+0000 7fa3d1ffb640 1 -- 172.21.15.55:0/1092613574 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa3c8001c60 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.715+0000 7fa3d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.715+0000 7fa3d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.715+0000 7fa3d1ffb640 10 monclient: dump: 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:07.720 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:07.721 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:07.721 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:07.721 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:07.721 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:07.721 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:07.721 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:07.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.715+0000 7fa3d3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:07.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.715+0000 7fa3d3fff640 4 set_mon_vals no callback set 2024-03-29T11:47:07.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.716+0000 7fa3d3fff640 10 monclient: shutdown 2024-03-29T11:47:07.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.716+0000 7fa3d3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:07.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.716+0000 7fa3d3fff640 1 -- 172.21.15.55:0/1092613574 >> v1:172.21.15.55:6790/0 conn(0x7fa3cc0c92b0 legacy=0x7fa3cc0d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:07.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.716+0000 7fa3d3fff640 1 -- 172.21.15.55:0/1092613574 shutdown_connections 2024-03-29T11:47:07.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.716+0000 7fa3d3fff640 1 -- 172.21.15.55:0/1092613574 >> 172.21.15.55:0/1092613574 conn(0x7fa3cc0058f0 msgr2=0x7fa3cc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:07.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.719+0000 7fa3d3fff640 1 -- 172.21.15.55:0/1092613574 shutdown_connections 2024-03-29T11:47:07.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.719+0000 7fa3d3fff640 1 -- 172.21.15.55:0/1092613574 wait complete. 2024-03-29T11:47:07.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.720+0000 7fa3d3fff640 10 monclient: build_initial_monmap 2024-03-29T11:47:07.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.720+0000 7fa3d3fff640 10 monclient: monmap: 2024-03-29T11:47:07.726 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:07.726 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:07.726 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:47:07.726 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:47:07.727 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:07.727 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:07.727 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:07.727 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:47:07.727 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:47:07.727 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:07.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.720+0000 7fa3d3fff640 1 librados: starting msgr at 2024-03-29T11:47:07.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.720+0000 7fa3d3fff640 1 librados: starting objecter 2024-03-29T11:47:07.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.721+0000 7fa3d3fff640 1 Processor -- start 2024-03-29T11:47:07.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.721+0000 7fa3d3fff640 1 -- start start 2024-03-29T11:47:07.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.721+0000 7fa3d3fff640 1 librados: setting wanted keys 2024-03-29T11:47:07.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.721+0000 7fa3d3fff640 1 librados: calling monclient init 2024-03-29T11:47:07.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.721+0000 7fa3d3fff640 10 monclient: init 2024-03-29T11:47:07.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.722+0000 7fa3d3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:07.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.722+0000 7fa3d3fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:47:07.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.723+0000 7fa3d3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa3cc0066e0 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:07.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.723+0000 7fa3d3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa3cc0c92b0 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:07.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.723+0000 7fa3d3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa3cc0cf820 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:07.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.723+0000 7fa3d3fff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc0cb020 con 0x7fa3cc0cf820 2024-03-29T11:47:07.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.723+0000 7fa3d3fff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc0cc740 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.723+0000 7fa3d3fff640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc0cca20 con 0x7fa3cc0066e0 2024-03-29T11:47:07.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.723+0000 7fa3d3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:07.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.723+0000 7fa3d3fff640 10 monclient(hunting): authenticate will time out at 1241.003501s 2024-03-29T11:47:07.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.727+0000 7fa3d27fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa3cc0c92b0 0x7fa3cc033910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:51573/0 (socket says 172.21.15.55:51573) 2024-03-29T11:47:07.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.727+0000 7fa3d27fc640 1 -- 172.21.15.55:0/3997178878 learned_addr learned my addr 172.21.15.55:0/3997178878 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:07.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.727+0000 7fa3bbfff640 10 client.?.objecter ms_handle_connect 0x7fa3cc0c92b0 2024-03-29T11:47:07.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.727+0000 7fa3bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:07.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.728+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436360663 0 0) 0x7fa3cc0cc740 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.734 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.728+0000 7fa3bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:47:07.734 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.728+0000 7fa3bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:07.734 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.728+0000 7fa3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:07.734 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.728+0000 7fa3bbfff640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:47:07.734 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.728+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c0004eb0 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.730+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3495009647 0 0) 0x7fa3c0004eb0 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.730+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3cc0cc740 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.730+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa3c40018b0 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.730+0000 7fa3bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:07.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.730+0000 7fa3bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.730+0000 7fa3bbfff640 10 monclient(hunting): dump: 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:07.736 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.730+0000 7fa3bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:07.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.732+0000 7fa3bbfff640 10 client.?.objecter ms_handle_connect 0x7fa3cc0066e0 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.732+0000 7fa3bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.732+0000 7fa3bbfff640 10 client.?.objecter ms_handle_connect 0x7fa3cc0cf820 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.732+0000 7fa3bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.732+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005677620 0 0) 0x7fa3cc0cca20 con 0x7fa3cc0066e0 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.732+0000 7fa3bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.732+0000 7fa3bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.732+0000 7fa3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.733+0000 7fa3bbfff640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.733+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c0004eb0 con 0x7fa3cc0066e0 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.733+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3956591475 0 0) 0x7fa3cc0cc740 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.733+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 >> v1:172.21.15.55:6790/0 conn(0x7fa3cc0066e0 legacy=0x7fa3cc0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.733+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 >> v1:172.21.15.55:6789/0 conn(0x7fa3cc0cf820 legacy=0x7fa3cc16b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:07.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.733+0000 7fa3bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:47:07.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.734+0000 7fa3bbfff640 1 monclient: found mon.b 2024-03-29T11:47:07.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.734+0000 7fa3bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:07.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.735+0000 7fa3bbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:07.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.735+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0cb5f0 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.735+0000 7fa3bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:47:07.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.735+0000 7fa3bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:47:07.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.735+0000 7fa3bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:07.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.735+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3c40035f0 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.735+0000 7fa3bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa3c40039c0 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3d3fff640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3d3fff640 10 monclient: _renew_subs 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3d3fff640 1 -- 172.21.15.55:0/3997178878 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc0cbb10 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3d3fff640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3d3fff640 10 monclient: _renew_subs 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3d3fff640 1 -- 172.21.15.55:0/3997178878 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc0cb3b0 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3d3fff640 1 librados: init done 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.736+0000 7fa3bbfff640 10 monclient: dump: 2024-03-29T11:47:07.743 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:07.744 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:07.744 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:07.744 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:07.744 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:07.744 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:07.744 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:07.744 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:07.744 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:07.744 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.738+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fa3c4003b20 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.739+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fa3c4084800 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.739+0000 7fa3d17fa640 4 set_mon_vals no callback set 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.739+0000 7fa3b9ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.739+0000 7fa3bbfff640 10 client.4405.objecter ms_dispatch 0x7fa3d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.740+0000 7fa3bbfff640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.740+0000 7fa3bbfff640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.740+0000 7fa3b9ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.740+0000 7fa3b9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:07.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.740+0000 7fa3bbfff640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T11:47:07.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.740+0000 7fa3b9ffb640 1 -- 172.21.15.55:0/3997178878 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b4005080 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.740+0000 7fa3d2ffd640 10 monclient: get_auth_request con 0x7fa3c006d9e0 auth_method 0 2024-03-29T11:47:07.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.740+0000 7fa3bbfff640 10 client.4405.objecter ms_handle_connect 0x7fa3c006d9e0 2024-03-29T11:47:07.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.748+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3c40500e0 con 0x7fa3cc0c92b0 2024-03-29T11:47:07.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.748+0000 7fa3bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:07.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:07.748+0000 7fa3bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:08.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.080+0000 7fa3b9ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:08.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.080+0000 7fa3b9ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:08.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.080+0000 7fa3b9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:08.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.080+0000 7fa3b9ffb640 1 -- 172.21.15.55:0/3997178878 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3b4005330 con 0x7fa3cc0c92b0 2024-03-29T11:47:08.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.081+0000 7fa3bbfff640 1 -- 172.21.15.55:0/3997178878 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fa3c4052010 con 0x7fa3cc0c92b0 2024-03-29T11:47:08.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.081+0000 7fa3bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:08.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.081+0000 7fa3bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:08.088 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:47:08.088 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 10 librados: watch_flush enter 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 10 librados: watch_flush exit 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 10 client.4405.objecter successfully canceled tick 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 1 -- 172.21.15.55:0/3997178878 >> v1:172.21.15.171:6816/1463197163 conn(0x7fa3c006d9e0 legacy=0x7fa3c006fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 10 monclient: shutdown 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 1 -- 172.21.15.55:0/3997178878 >> v1:172.21.15.171:6789/0 conn(0x7fa3cc0c92b0 legacy=0x7fa3cc033910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 1 -- 172.21.15.55:0/3997178878 shutdown_connections 2024-03-29T11:47:08.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.086+0000 7fa3b9ffb640 1 -- 172.21.15.55:0/3997178878 >> 172.21.15.55:0/3997178878 conn(0x7fa3cc0058f0 msgr2=0x7fa3cc0c2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:08.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.088+0000 7fa3b9ffb640 1 -- 172.21.15.55:0/3997178878 shutdown_connections 2024-03-29T11:47:08.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.088+0000 7fa3b9ffb640 1 -- 172.21.15.55:0/3997178878 wait complete. 2024-03-29T11:47:08.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:08.088+0000 7fa3b9ffb640 1 librados: shutdown 2024-03-29T11:47:08.111 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:14.917 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:15.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.073+0000 7f9af987d640 10 monclient: get_monmap_and_config 2024-03-29T11:47:15.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.073+0000 7f9af987d640 10 monclient: build_initial_monmap 2024-03-29T11:47:15.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.073+0000 7f9af987d640 10 monclient: monmap: 2024-03-29T11:47:15.081 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:15.082 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:15.082 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:47:15.074741+0000 2024-03-29T11:47:15.082 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:47:15.074741+0000 2024-03-29T11:47:15.082 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:15.082 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:15.082 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:15.082 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:47:15.083 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:47:15.083 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:15.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.075+0000 7f9af987d640 1 Processor -- start 2024-03-29T11:47:15.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.075+0000 7f9af987d640 1 -- start start 2024-03-29T11:47:15.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.075+0000 7f9af987d640 10 monclient: init 2024-03-29T11:47:15.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:15.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:47:15.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 10 monclient(hunting): picked mon.noname-a con 0x7f9af40b0770 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 10 monclient(hunting): picked mon.noname-b con 0x7f9af414a700 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 10 monclient(hunting): picked mon.noname-c con 0x7f9af40ac5a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af40aba70 con 0x7f9af40b0770 2024-03-29T11:47:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af40aecb0 con 0x7f9af40ac5a0 2024-03-29T11:47:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af40af150 con 0x7f9af414a700 2024-03-29T11:47:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:15.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.076+0000 7f9af987d640 10 monclient(hunting): authenticate will time out at 1248.355826s 2024-03-29T11:47:15.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.079+0000 7f9af37fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9af40ac5a0 0x7f9af4151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:55105/0 (socket says 172.21.15.55:55105) 2024-03-29T11:47:15.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.079+0000 7f9af37fe640 1 -- 172.21.15.55:0/1828214744 learned_addr learned my addr 172.21.15.55:0/1828214744 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:15.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033688106 0 0) 0x7f9af40aecb0 con 0x7f9af40ac5a0 2024-03-29T11:47:15.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:47:15.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:15.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:15.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc003520 con 0x7f9af40ac5a0 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542752996 0 0) 0x7f9af40af150 con 0x7f9af414a700 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.083+0000 7f9af1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af40aecb0 con 0x7f9af414a700 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 548726343 0 0) 0x7f9af40aecb0 con 0x7f9af414a700 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 >> v1:172.21.15.171:6789/0 conn(0x7f9af40ac5a0 legacy=0x7f9af4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 >> v1:172.21.15.55:6789/0 conn(0x7f9af40b0770 legacy=0x7f9af4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af40ac9d0 con 0x7f9af414a700 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9aec0019f0 con 0x7f9af414a700 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.084+0000 7f9af1ffb640 10 monclient: dump: 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:15.094 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af987d640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af987d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aec001c10 con 0x7f9af414a700 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af987d640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af987d640 4 set_mon_vals no callback set 2024-03-29T11:47:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af1ffb640 1 -- 172.21.15.55:0/1828214744 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9aec001fa0 con 0x7f9af414a700 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.085+0000 7f9af1ffb640 10 monclient: dump: 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.086+0000 7f9af987d640 10 monclient: shutdown 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.086+0000 7f9af987d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.086+0000 7f9af987d640 1 -- 172.21.15.55:0/1828214744 >> v1:172.21.15.55:6790/0 conn(0x7f9af414a700 legacy=0x7f9af40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:15.097 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.086+0000 7f9af987d640 1 -- 172.21.15.55:0/1828214744 shutdown_connections 2024-03-29T11:47:15.097 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.086+0000 7f9af987d640 1 -- 172.21.15.55:0/1828214744 >> 172.21.15.55:0/1828214744 conn(0x7f9af406b660 msgr2=0x7f9af406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:15.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.087+0000 7f9af987d640 1 -- 172.21.15.55:0/1828214744 shutdown_connections 2024-03-29T11:47:15.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.089+0000 7f9af987d640 1 -- 172.21.15.55:0/1828214744 wait complete. 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.090+0000 7f9af987d640 10 monclient: build_initial_monmap 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.090+0000 7f9af987d640 10 monclient: monmap: 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.091+0000 7f9af987d640 1 librados: starting msgr at 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.091+0000 7f9af987d640 1 librados: starting objecter 2024-03-29T11:47:15.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.091+0000 7f9af987d640 1 Processor -- start 2024-03-29T11:47:15.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.092+0000 7f9af987d640 1 -- start start 2024-03-29T11:47:15.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.092+0000 7f9af987d640 1 librados: setting wanted keys 2024-03-29T11:47:15.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.092+0000 7f9af987d640 1 librados: calling monclient init 2024-03-29T11:47:15.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.092+0000 7f9af987d640 10 monclient: init 2024-03-29T11:47:15.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.093+0000 7f9af987d640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:15.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.093+0000 7f9af987d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:47:15.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.093+0000 7f9af987d640 10 monclient(hunting): picked mon.noname-a con 0x7f9af40ac5a0 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:15.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.095+0000 7f9af987d640 10 monclient(hunting): picked mon.noname-b con 0x7f9af40b0770 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:15.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.095+0000 7f9af987d640 10 monclient(hunting): picked mon.noname-c con 0x7f9af414a700 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:15.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.099+0000 7f9af987d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af4128650 con 0x7f9af40ac5a0 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.099+0000 7f9af987d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af41db7f0 con 0x7f9af40b0770 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.099+0000 7f9af987d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af41dc420 con 0x7f9af414a700 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.099+0000 7f9af987d640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.099+0000 7f9af987d640 10 monclient(hunting): authenticate will time out at 1248.379070s 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.099+0000 7f9af27fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9af40b0770 0x7f9af41dbd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:55205/0 (socket says 172.21.15.55:55205) 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.099+0000 7f9af27fc640 1 -- 172.21.15.55:0/2465736990 learned_addr learned my addr 172.21.15.55:0/2465736990 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.099+0000 7f9adbfff640 10 client.?.objecter ms_handle_connect 0x7f9af40b0770 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.099+0000 7f9adbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 10 client.?.objecter ms_handle_connect 0x7f9af414a700 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280776636 0 0) 0x7f9af41db7f0 con 0x7f9af40b0770 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:15.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:47:15.109 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae0004eb0 con 0x7f9af40b0770 2024-03-29T11:47:15.109 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 10 client.?.objecter ms_handle_connect 0x7f9af40ac5a0 2024-03-29T11:47:15.109 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.100+0000 7f9adbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:15.110 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398005456 0 0) 0x7f9af4128650 con 0x7f9af40ac5a0 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af41db7f0 con 0x7f9af40ac5a0 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1948686404 0 0) 0x7f9ae0004eb0 con 0x7f9af40b0770 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9af4128650 con 0x7f9af40b0770 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093552103 0 0) 0x7f9af41dc420 con 0x7f9af414a700 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:15.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:47:15.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae0004eb0 con 0x7f9af414a700 2024-03-29T11:47:15.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9aec001fe0 con 0x7f9af40b0770 2024-03-29T11:47:15.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:15.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:15.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): dump: 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.103+0000 7f9adbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3063297212 0 0) 0x7f9af4128650 con 0x7f9af40b0770 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 >> v1:172.21.15.55:6790/0 conn(0x7f9af414a700 legacy=0x7f9af4145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:15.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 >> v1:172.21.15.55:6789/0 conn(0x7f9af40ac5a0 legacy=0x7f9af41261f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 1 monclient: found mon.b 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af41dc5b0 con 0x7f9af40b0770 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 10 monclient: _finish_auth 0 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9af987d640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9af987d640 10 monclient: _renew_subs 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9af987d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af41239d0 con 0x7f9af40b0770 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9af987d640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9af987d640 10 monclient: _renew_subs 2024-03-29T11:47:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9af987d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af4145970 con 0x7f9af40b0770 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9af987d640 1 librados: init done 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aec003c20 con 0x7f9af40b0770 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9aec003e90 con 0x7f9af40b0770 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.104+0000 7f9adbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.105+0000 7f9adbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.105+0000 7f9adbfff640 10 monclient: dump: 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:15.120 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.105+0000 7f9af17fa640 4 set_mon_vals no callback set 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.105+0000 7f9af987d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.106+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f9aec003ff0 con 0x7f9af40b0770 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.107+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f9aec084c80 con 0x7f9af40b0770 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.107+0000 7f9adbfff640 10 client.4411.objecter ms_dispatch 0x7f9af4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.107+0000 7f9adbfff640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.107+0000 7f9adbfff640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.107+0000 7f9adbfff640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.107+0000 7f9af987d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.107+0000 7f9af987d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:15.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.108+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad4005080 con 0x7f9af40b0770 2024-03-29T11:47:15.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.108+0000 7f9af2ffd640 10 monclient: get_auth_request con 0x7f9ae006dc10 auth_method 0 2024-03-29T11:47:15.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.108+0000 7f9adbfff640 10 client.4411.objecter ms_handle_connect 0x7f9ae006dc10 2024-03-29T11:47:15.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.117+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9aec084e60 con 0x7f9af40b0770 2024-03-29T11:47:15.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.117+0000 7f9adbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:15.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.117+0000 7f9adbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:15.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.461+0000 7f9af987d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:15.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.461+0000 7f9af987d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:15.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.461+0000 7f9af987d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:15.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.461+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ad4005330 con 0x7f9af40b0770 2024-03-29T11:47:15.471 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:47:15.471 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:15.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.464+0000 7f9adbfff640 1 -- 172.21.15.55:0/2465736990 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f9aec050560 con 0x7f9af40b0770 2024-03-29T11:47:15.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.464+0000 7f9adbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:15.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.464+0000 7f9adbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:15.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.468+0000 7f9af987d640 10 librados: watch_flush enter 2024-03-29T11:47:15.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.468+0000 7f9af987d640 10 librados: watch_flush exit 2024-03-29T11:47:15.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.468+0000 7f9af987d640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:47:15.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.468+0000 7f9af987d640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:47:15.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.471+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 >> v1:172.21.15.171:6816/1463197163 conn(0x7f9ae006dc10 legacy=0x7f9ae00700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:15.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.471+0000 7f9af987d640 10 monclient: shutdown 2024-03-29T11:47:15.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.471+0000 7f9af987d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:15.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.471+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 >> v1:172.21.15.171:6789/0 conn(0x7f9af40b0770 legacy=0x7f9af41dbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:15.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.471+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 shutdown_connections 2024-03-29T11:47:15.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.471+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 >> 172.21.15.55:0/2465736990 conn(0x7f9af406b660 msgr2=0x7f9af40b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:15.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.472+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 shutdown_connections 2024-03-29T11:47:15.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.472+0000 7f9af987d640 1 -- 172.21.15.55:0/2465736990 wait complete. 2024-03-29T11:47:15.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:15.472+0000 7f9af987d640 1 librados: shutdown 2024-03-29T11:47:15.499 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:22.306 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.456+0000 7f59ef4a8640 10 monclient: get_monmap_and_config 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.456+0000 7f59ef4a8640 10 monclient: build_initial_monmap 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.456+0000 7f59ef4a8640 10 monclient: monmap: 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:47:22.457461+0000 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:47:22.457461+0000 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:22.463 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:47:22.464 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:47:22.464 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:22.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.457+0000 7f59ef4a8640 1 Processor -- start 2024-03-29T11:47:22.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.457+0000 7f59ef4a8640 1 -- start start 2024-03-29T11:47:22.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.457+0000 7f59ef4a8640 10 monclient: init 2024-03-29T11:47:22.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f59e80ae250 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f59e813f970 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f59e8142800 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e80afe00 con 0x7f59e8142800 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8142c10 con 0x7f59e80ae250 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e80afb20 con 0x7f59e813f970 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.458+0000 7f59ef4a8640 10 monclient(hunting): authenticate will time out at 1255.738287s 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.462+0000 7f59eca43640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f59e813f970 0x7f59e81420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:41363/0 (socket says 172.21.15.55:41363) 2024-03-29T11:47:22.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.462+0000 7f59eca43640 1 -- 172.21.15.55:0/1144702411 learned_addr learned my addr 172.21.15.55:0/1144702411 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:22.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.463+0000 7f59dffff640 1 -- 172.21.15.55:0/1144702411 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772946436 0 0) 0x7f59e80afb20 con 0x7f59e813f970 2024-03-29T11:47:22.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.463+0000 7f59dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:47:22.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.463+0000 7f59dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:22.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.463+0000 7f59dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:22.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.463+0000 7f59dffff640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:47:22.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.463+0000 7f59dffff640 1 -- 172.21.15.55:0/1144702411 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d4003520 con 0x7f59e813f970 2024-03-29T11:47:22.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.463+0000 7f59dffff640 1 -- 172.21.15.55:0/1144702411 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2820427668 0 0) 0x7f59d4003520 con 0x7f59e813f970 2024-03-29T11:47:22.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.463+0000 7f59dffff640 1 -- 172.21.15.55:0/1144702411 >> v1:172.21.15.171:6789/0 conn(0x7f59e80ae250 legacy=0x7f59e813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:47:22.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.463+0000 7f59dffff640 1 -- 172.21.15.55:0/1144702411 >> v1:172.21.15.55:6789/0 conn(0x7f59e8142800 legacy=0x7f59e814da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:47:22.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:47:22.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 1 monclient: found mon.noname-b 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 1 -- 172.21.15.55:0/1144702411 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e80aab80 con 0x7f59e813f970 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 10 monclient: _finish_auth 0 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 1 -- 172.21.15.55:0/1144702411 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f59d8001670 con 0x7f59e813f970 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.464+0000 7f59dffff640 10 monclient: dump: 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.472 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.465+0000 7f59ef4a8640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.465+0000 7f59ef4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.468+0000 7f59dffff640 1 -- 172.21.15.55:0/1144702411 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59d80018d0 con 0x7f59e813f970 2024-03-29T11:47:22.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.468+0000 7f59dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.468+0000 7f59dffff640 1 -- 172.21.15.55:0/1144702411 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f59d8001d20 con 0x7f59e813f970 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.468+0000 7f59dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.468+0000 7f59dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.468+0000 7f59dffff640 10 monclient: dump: 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:22.474 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:22.475 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.468+0000 7f59ef4a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:22.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.468+0000 7f59ef4a8640 4 set_mon_vals no callback set 2024-03-29T11:47:22.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.469+0000 7f59ef4a8640 10 monclient: shutdown 2024-03-29T11:47:22.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.469+0000 7f59ef4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:22.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.469+0000 7f59ef4a8640 1 -- 172.21.15.55:0/1144702411 >> v1:172.21.15.55:6790/0 conn(0x7f59e813f970 legacy=0x7f59e81420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:22.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.469+0000 7f59ef4a8640 1 -- 172.21.15.55:0/1144702411 shutdown_connections 2024-03-29T11:47:22.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.469+0000 7f59ef4a8640 1 -- 172.21.15.55:0/1144702411 >> 172.21.15.55:0/1144702411 conn(0x7f59e80b35e0 msgr2=0x7f59e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:22.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.473+0000 7f59ef4a8640 1 -- 172.21.15.55:0/1144702411 shutdown_connections 2024-03-29T11:47:22.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.474+0000 7f59ef4a8640 1 -- 172.21.15.55:0/1144702411 wait complete. 2024-03-29T11:47:22.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.476+0000 7f59ef4a8640 10 monclient: build_initial_monmap 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.476+0000 7f59ef4a8640 10 monclient: monmap: 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.477+0000 7f59ef4a8640 1 librados: starting msgr at 2024-03-29T11:47:22.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.477+0000 7f59ef4a8640 1 librados: starting objecter 2024-03-29T11:47:22.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.477+0000 7f59ef4a8640 1 Processor -- start 2024-03-29T11:47:22.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.477+0000 7f59ef4a8640 1 -- start start 2024-03-29T11:47:22.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.477+0000 7f59ef4a8640 1 librados: setting wanted keys 2024-03-29T11:47:22.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.477+0000 7f59ef4a8640 1 librados: calling monclient init 2024-03-29T11:47:22.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.477+0000 7f59ef4a8640 10 monclient: init 2024-03-29T11:47:22.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:22.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:47:22.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f59e80ae250 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:22.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f59e8142800 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:22.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f59e80ac970 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:22.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8143b60 con 0x7f59e8142800 2024-03-29T11:47:22.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8143e40 con 0x7f59e80ae250 2024-03-29T11:47:22.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8144120 con 0x7f59e80ac970 2024-03-29T11:47:22.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:22.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.478+0000 7f59ef4a8640 10 monclient(hunting): authenticate will time out at 1255.757985s 2024-03-29T11:47:22.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.484+0000 7f59ed244640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f59e80ae250 0x7f59e8139870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:59167/0 (socket says 172.21.15.55:59167) 2024-03-29T11:47:22.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.484+0000 7f59ed244640 1 -- 172.21.15.55:0/2798299482 learned_addr learned my addr 172.21.15.55:0/2798299482 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:22.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.484+0000 7f59ddffb640 10 client.?.objecter ms_handle_connect 0x7f59e8142800 2024-03-29T11:47:22.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.484+0000 7f59ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:22.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318141881 0 0) 0x7f59e8143b60 con 0x7f59e8142800 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc004ee0 con 0x7f59e8142800 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 client.?.objecter ms_handle_connect 0x7f59e80ac970 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 client.?.objecter ms_handle_connect 0x7f59e80ae250 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058505960 0 0) 0x7f59e8144120 con 0x7f59e80ac970 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:47:22.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:22.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:22.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:47:22.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e8143b60 con 0x7f59e80ac970 2024-03-29T11:47:22.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687798734 0 0) 0x7f59e8143e40 con 0x7f59e80ae250 2024-03-29T11:47:22.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:47:22.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:22.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:22.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:47:22.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.488+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e8144120 con 0x7f59e80ae250 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4118035548 0 0) 0x7f59e8144120 con 0x7f59e80ae250 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59e8143e40 con 0x7f59e80ae250 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f59e40018a0 con 0x7f59e80ae250 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): dump: 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.497 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 976192407 0 0) 0x7f59e8143b60 con 0x7f59e80ac970 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59e8144120 con 0x7f59e80ac970 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f59e0001a70 con 0x7f59e80ac970 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): dump: 2024-03-29T11:47:22.498 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3482762798 0 0) 0x7f59cc004ee0 con 0x7f59e8142800 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59e8143b60 con 0x7f59e8142800 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f59d8001630 con 0x7f59e8142800 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:47:22.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.489+0000 7f59ddffb640 10 monclient(hunting): dump: 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1301439599 0 0) 0x7f59e8144120 con 0x7f59e80ac970 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 >> v1:172.21.15.171:6789/0 conn(0x7f59e80ae250 legacy=0x7f59e8139870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 >> v1:172.21.15.55:6789/0 conn(0x7f59e8142800 legacy=0x7f59e81e3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:22.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:47:22.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 1 monclient: found mon.c 2024-03-29T11:47:22.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:22.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:22.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e8144400 con 0x7f59e80ac970 2024-03-29T11:47:22.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:47:22.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:47:22.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.490+0000 7f59ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:22.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.491+0000 7f59ef4a8640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:47:22.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.491+0000 7f59ef4a8640 10 monclient: _renew_subs 2024-03-29T11:47:22.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.491+0000 7f59ef4a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:22.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.491+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e8144c50 con 0x7f59e80ac970 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ef4a8640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ef4a8640 10 monclient: _renew_subs 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ef4a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e8144a00 con 0x7f59e80ac970 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ef4a8640 1 librados: init done 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59e0001f60 con 0x7f59e80ac970 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f59e0003b10 con 0x7f59e80ac970 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59ddffb640 10 monclient: dump: 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:22.503 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.492+0000 7f59df7fe640 4 set_mon_vals no callback set 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.495+0000 7f59ef4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.495+0000 7f59ef4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.495+0000 7f59ef4a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.495+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c8005080 con 0x7f59e80ac970 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.495+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f59e0019ae0 con 0x7f59e80ac970 2024-03-29T11:47:22.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.496+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f59e00839b0 con 0x7f59e80ac970 2024-03-29T11:47:22.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.496+0000 7f59ddffb640 10 client.4436.objecter ms_dispatch 0x7f59e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:47:22.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.496+0000 7f59ddffb640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:47:22.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.496+0000 7f59ddffb640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:47:22.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.496+0000 7f59ddffb640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T11:47:22.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.498+0000 7f59ed244640 10 monclient: get_auth_request con 0x7f59cc06e170 auth_method 0 2024-03-29T11:47:22.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.499+0000 7f59ddffb640 10 client.4436.objecter ms_handle_connect 0x7f59cc06e170 2024-03-29T11:47:22.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.503+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59e004f290 con 0x7f59e80ac970 2024-03-29T11:47:22.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.503+0000 7f59ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:22.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.503+0000 7f59ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:22.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.830+0000 7f59ef4a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:22.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.830+0000 7f59ef4a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:22.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.830+0000 7f59ef4a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:22.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.830+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59c8005330 con 0x7f59e80ac970 2024-03-29T11:47:22.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.831+0000 7f59ddffb640 1 -- 172.21.15.55:0/2798299482 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f59e00511c0 con 0x7f59e80ac970 2024-03-29T11:47:22.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.831+0000 7f59ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:22.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.831+0000 7f59ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:22.839 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:47:22.839 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:22.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 10 librados: watch_flush enter 2024-03-29T11:47:22.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 10 librados: watch_flush exit 2024-03-29T11:47:22.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T11:47:22.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 10 client.4436.objecter successfully canceled tick 2024-03-29T11:47:22.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 >> v1:172.21.15.171:6816/1463197163 conn(0x7f59cc06e170 legacy=0x7f59cc070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:22.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 10 monclient: shutdown 2024-03-29T11:47:22.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:22.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 >> v1:172.21.15.55:6790/0 conn(0x7f59e80ac970 legacy=0x7f59e81e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:22.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 shutdown_connections 2024-03-29T11:47:22.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.837+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 >> 172.21.15.55:0/2798299482 conn(0x7f59e80b35e0 msgr2=0x7f59e813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:22.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.838+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 shutdown_connections 2024-03-29T11:47:22.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.838+0000 7f59ef4a8640 1 -- 172.21.15.55:0/2798299482 wait complete. 2024-03-29T11:47:22.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:22.838+0000 7f59ef4a8640 1 librados: shutdown 2024-03-29T11:47:22.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:29.671 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:29.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.818+0000 7fd2de655640 10 monclient: get_monmap_and_config 2024-03-29T11:47:29.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.818+0000 7fd2de655640 10 monclient: build_initial_monmap 2024-03-29T11:47:29.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.818+0000 7fd2de655640 10 monclient: monmap: 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:47:29.819219+0000 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:47:29.819219+0000 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:47:29.824 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:29.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.821+0000 7fd2de655640 1 Processor -- start 2024-03-29T11:47:29.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.821+0000 7fd2de655640 1 -- start start 2024-03-29T11:47:29.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.821+0000 7fd2de655640 10 monclient: init 2024-03-29T11:47:29.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d81457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d813f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d80a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80aaef0 con 0x7fd2d81457f0 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d8145bd0 con 0x7fd2d80a9d70 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d8145eb0 con 0x7fd2d813f650 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.822+0000 7fd2de655640 10 monclient(hunting): authenticate will time out at 1263.102246s 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.828+0000 7fd2dcbf2640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd2d80a9d70 0x7fd2d814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:44143/0 (socket says 172.21.15.55:44143) 2024-03-29T11:47:29.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.828+0000 7fd2dcbf2640 1 -- 172.21.15.55:0/3588836277 learned_addr learned my addr 172.21.15.55:0/3588836277 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:29.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.829+0000 7fd2d6ffd640 1 -- 172.21.15.55:0/3588836277 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466771837 0 0) 0x7fd2d8145eb0 con 0x7fd2d813f650 2024-03-29T11:47:29.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.829+0000 7fd2d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:29.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.829+0000 7fd2d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:29.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.829+0000 7fd2d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:29.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.829+0000 7fd2d6ffd640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:47:29.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.829+0000 7fd2d6ffd640 1 -- 172.21.15.55:0/3588836277 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c0003520 con 0x7fd2d813f650 2024-03-29T11:47:29.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.829+0000 7fd2d6ffd640 1 -- 172.21.15.55:0/3588836277 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3839574094 0 0) 0x7fd2c0003520 con 0x7fd2d813f650 2024-03-29T11:47:29.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.829+0000 7fd2d6ffd640 1 -- 172.21.15.55:0/3588836277 >> v1:172.21.15.171:6789/0 conn(0x7fd2d80a9d70 legacy=0x7fd2d814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:47:29.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.829+0000 7fd2d6ffd640 1 -- 172.21.15.55:0/3588836277 >> v1:172.21.15.55:6789/0 conn(0x7fd2d81457f0 legacy=0x7fd2d813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:47:29.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:47:29.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 1 -- 172.21.15.55:0/3588836277 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d8146340 con 0x7fd2d813f650 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 1 -- 172.21.15.55:0/3588836277 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd2cc001690 con 0x7fd2d813f650 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2d6ffd640 10 monclient: dump: 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:29.839 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2de655640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.832+0000 7fd2de655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.833+0000 7fd2d6ffd640 1 -- 172.21.15.55:0/3588836277 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2cc001860 con 0x7fd2d813f650 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.833+0000 7fd2d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.833+0000 7fd2d6ffd640 1 -- 172.21.15.55:0/3588836277 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd2cc001cb0 con 0x7fd2d813f650 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.833+0000 7fd2d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.833+0000 7fd2d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.833+0000 7fd2d6ffd640 10 monclient: dump: 2024-03-29T11:47:29.840 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:29.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.833+0000 7fd2de655640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:29.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.833+0000 7fd2de655640 4 set_mon_vals no callback set 2024-03-29T11:47:29.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.834+0000 7fd2de655640 10 monclient: shutdown 2024-03-29T11:47:29.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.834+0000 7fd2de655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:29.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.834+0000 7fd2de655640 1 -- 172.21.15.55:0/3588836277 >> v1:172.21.15.55:6790/0 conn(0x7fd2d813f650 legacy=0x7fd2d80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:29.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.834+0000 7fd2de655640 1 -- 172.21.15.55:0/3588836277 shutdown_connections 2024-03-29T11:47:29.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.834+0000 7fd2de655640 1 -- 172.21.15.55:0/3588836277 >> 172.21.15.55:0/3588836277 conn(0x7fd2d8066a00 msgr2=0x7fd2d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:29.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.836+0000 7fd2de655640 1 -- 172.21.15.55:0/3588836277 shutdown_connections 2024-03-29T11:47:29.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.836+0000 7fd2de655640 1 -- 172.21.15.55:0/3588836277 wait complete. 2024-03-29T11:47:29.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.837+0000 7fd2de655640 10 monclient: build_initial_monmap 2024-03-29T11:47:29.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.837+0000 7fd2de655640 10 monclient: monmap: 2024-03-29T11:47:29.844 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.838+0000 7fd2de655640 1 librados: starting msgr at 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.838+0000 7fd2de655640 1 librados: starting objecter 2024-03-29T11:47:29.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.838+0000 7fd2de655640 1 Processor -- start 2024-03-29T11:47:29.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.838+0000 7fd2de655640 1 -- start start 2024-03-29T11:47:29.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.838+0000 7fd2de655640 1 librados: setting wanted keys 2024-03-29T11:47:29.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.838+0000 7fd2de655640 1 librados: calling monclient init 2024-03-29T11:47:29.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.838+0000 7fd2de655640 10 monclient: init 2024-03-29T11:47:29.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:29.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:47:29.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d80a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:29.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d813f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:29.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d81457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:29.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d8071dc0 con 0x7fd2d813f650 2024-03-29T11:47:29.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d8131c60 con 0x7fd2d81457f0 2024-03-29T11:47:29.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d8071870 con 0x7fd2d80a9d70 2024-03-29T11:47:29.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:29.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.839+0000 7fd2de655640 10 monclient(hunting): authenticate will time out at 1263.119149s 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.843+0000 7fd2dcbf2640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd2d81457f0 0x7fd2d8132b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:44265/0 (socket says 172.21.15.55:44265) 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.843+0000 7fd2dcbf2640 1 -- 172.21.15.55:0/784663095 learned_addr learned my addr 172.21.15.55:0/784663095 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.843+0000 7fd2d4ff9640 10 client.?.objecter ms_handle_connect 0x7fd2d81457f0 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.843+0000 7fd2d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522414222 0 0) 0x7fd2d8131c60 con 0x7fd2d81457f0 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:47:29.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4004eb0 con 0x7fd2d81457f0 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2851491827 0 0) 0x7fd2c4004eb0 con 0x7fd2d81457f0 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2d8131c60 con 0x7fd2d81457f0 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd2d0001890 con 0x7fd2d81457f0 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 10 monclient(hunting): dump: 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:29.851 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:29.852 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:29.852 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:29.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:29.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2344377234 0 0) 0x7fd2d8131c60 con 0x7fd2d81457f0 2024-03-29T11:47:29.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.844+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 >> v1:172.21.15.55:6790/0 conn(0x7fd2d80a9d70 legacy=0x7fd2d8070cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:47:29.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.846+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 >> v1:172.21.15.55:6789/0 conn(0x7fd2d813f650 legacy=0x7fd2d81323f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:47:29.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2d4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:47:29.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2d4ff9640 1 monclient: found mon.b 2024-03-29T11:47:29.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:29.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2d4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d8071a00 con 0x7fd2d81457f0 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2d4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2d4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2de655640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2de655640 10 monclient: _renew_subs 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2de655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d81e72d0 con 0x7fd2d81457f0 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2de655640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2de655640 10 monclient: _renew_subs 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2de655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d81e7640 con 0x7fd2d81457f0 2024-03-29T11:47:29.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.847+0000 7fd2de655640 1 librados: init done 2024-03-29T11:47:29.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.850+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2d00019f0 con 0x7fd2d81457f0 2024-03-29T11:47:29.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.850+0000 7fd2d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:29.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.851+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd2d0004ea0 con 0x7fd2d81457f0 2024-03-29T11:47:29.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.851+0000 7fd2d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:29.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.851+0000 7fd2d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:47:29.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.851+0000 7fd2d4ff9640 10 monclient: dump: 2024-03-29T11:47:29.857 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:29.857 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:29.857 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.851+0000 7fd2d67fc640 4 set_mon_vals no callback set 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.851+0000 7fd2de655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.852+0000 7fd2de655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.852+0000 7fd2de655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:29.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.852+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b8005080 con 0x7fd2d81457f0 2024-03-29T11:47:29.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.853+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fd2d001ae70 con 0x7fd2d81457f0 2024-03-29T11:47:29.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.853+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fd2d0084e20 con 0x7fd2d81457f0 2024-03-29T11:47:29.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.853+0000 7fd2d4ff9640 10 client.4417.objecter ms_dispatch 0x7fd2d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:47:29.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.853+0000 7fd2d4ff9640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:47:29.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.853+0000 7fd2d4ff9640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:47:29.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.853+0000 7fd2d4ff9640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-29T11:47:29.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.855+0000 7fd2d7fff640 10 monclient: get_auth_request con 0x7fd2c406dc90 auth_method 0 2024-03-29T11:47:29.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.855+0000 7fd2d4ff9640 10 client.4417.objecter ms_handle_connect 0x7fd2c406dc90 2024-03-29T11:47:29.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.859+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2d0050700 con 0x7fd2d81457f0 2024-03-29T11:47:29.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.859+0000 7fd2d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:29.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:29.859+0000 7fd2d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:30.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.196+0000 7fd2de655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:30.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.196+0000 7fd2de655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:30.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.196+0000 7fd2de655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:30.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.196+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2b8005330 con 0x7fd2d81457f0 2024-03-29T11:47:30.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.197+0000 7fd2d4ff9640 1 -- 172.21.15.55:0/784663095 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fd2d0052630 con 0x7fd2d81457f0 2024-03-29T11:47:30.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.197+0000 7fd2d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:30.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.197+0000 7fd2d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:30.203 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:47:30.203 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:30.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.200+0000 7fd2de655640 10 librados: watch_flush enter 2024-03-29T11:47:30.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.200+0000 7fd2de655640 10 librados: watch_flush exit 2024-03-29T11:47:30.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.200+0000 7fd2de655640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-29T11:47:30.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.200+0000 7fd2de655640 10 client.4417.objecter successfully canceled tick 2024-03-29T11:47:30.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.201+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 >> v1:172.21.15.171:6816/1463197163 conn(0x7fd2c406dc90 legacy=0x7fd2c4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:30.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.201+0000 7fd2de655640 10 monclient: shutdown 2024-03-29T11:47:30.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.201+0000 7fd2de655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:30.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.201+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 >> v1:172.21.15.171:6789/0 conn(0x7fd2d81457f0 legacy=0x7fd2d8132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:30.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.202+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 shutdown_connections 2024-03-29T11:47:30.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.202+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 >> 172.21.15.55:0/784663095 conn(0x7fd2d8066a00 msgr2=0x7fd2d80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:30.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.203+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 shutdown_connections 2024-03-29T11:47:30.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.203+0000 7fd2de655640 1 -- 172.21.15.55:0/784663095 wait complete. 2024-03-29T11:47:30.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:30.204+0000 7fd2de655640 1 librados: shutdown 2024-03-29T11:47:30.230 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:37.037 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:37.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.187+0000 7fa140c7b640 10 monclient: get_monmap_and_config 2024-03-29T11:47:37.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.187+0000 7fa140c7b640 10 monclient: build_initial_monmap 2024-03-29T11:47:37.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.187+0000 7fa140c7b640 10 monclient: monmap: 2024-03-29T11:47:37.192 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:37.193 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:37.193 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:47:37.188106+0000 2024-03-29T11:47:37.193 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:47:37.188106+0000 2024-03-29T11:47:37.193 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:37.193 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:37.193 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:37.193 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:47:37.193 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:47:37.193 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:37.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.190+0000 7fa140c7b640 1 Processor -- start 2024-03-29T11:47:37.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.193+0000 7fa140c7b640 1 -- start start 2024-03-29T11:47:37.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.193+0000 7fa140c7b640 10 monclient: init 2024-03-29T11:47:37.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa13c1457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa13c13f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa13c0a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c0aaef0 con 0x7fa13c13f650 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c145bd0 con 0x7fa13c0a9d70 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c145eb0 con 0x7fa13c1457f0 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.194+0000 7fa140c7b640 10 monclient(hunting): authenticate will time out at 1270.473479s 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.197+0000 7fa139d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa13c13f650 0x7fa13c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56593/0 (socket says 172.21.15.55:56593) 2024-03-29T11:47:37.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.197+0000 7fa139d9b640 1 -- 172.21.15.55:0/889934629 learned_addr learned my addr 172.21.15.55:0/889934629 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.198+0000 7fa13959a640 1 -- 172.21.15.55:0/889934629 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743163816 0 0) 0x7fa13c0aaef0 con 0x7fa13c13f650 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.198+0000 7fa13959a640 10 monclient(hunting): _negotiate 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.198+0000 7fa13959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.198+0000 7fa13959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.198+0000 7fa13959a640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.198+0000 7fa13959a640 1 -- 172.21.15.55:0/889934629 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa124003520 con 0x7fa13c13f650 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.198+0000 7fa13959a640 1 -- 172.21.15.55:0/889934629 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3337234 0 0) 0x7fa124003520 con 0x7fa13c13f650 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.198+0000 7fa13959a640 1 -- 172.21.15.55:0/889934629 >> v1:172.21.15.55:6790/0 conn(0x7fa13c1457f0 legacy=0x7fa13c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 1 -- 172.21.15.55:0/889934629 >> v1:172.21.15.171:6789/0 conn(0x7fa13c0a9d70 legacy=0x7fa13c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 10 monclient: _finish_hunting 0 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 1 monclient: found mon.noname-a 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 1 -- 172.21.15.55:0/889934629 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c146340 con 0x7fa13c13f650 2024-03-29T11:47:37.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 10 monclient: _finish_auth 0 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 10 monclient: _check_auth_tickets 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 1 -- 172.21.15.55:0/889934629 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa12c001670 con 0x7fa13c13f650 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa13959a640 10 monclient: dump: 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:37.211 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:37.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa140c7b640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:47:37.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.201+0000 7fa140c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:37.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.203+0000 7fa13959a640 1 -- 172.21.15.55:0/889934629 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa12c0018d0 con 0x7fa13c13f650 2024-03-29T11:47:37.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.203+0000 7fa13959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.203+0000 7fa13959a640 1 -- 172.21.15.55:0/889934629 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa12c001d20 con 0x7fa13c13f650 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.203+0000 7fa13959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.203+0000 7fa13959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.203+0000 7fa13959a640 10 monclient: dump: 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:37.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.204+0000 7fa140c7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:37.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.204+0000 7fa140c7b640 4 set_mon_vals no callback set 2024-03-29T11:47:37.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.205+0000 7fa140c7b640 10 monclient: shutdown 2024-03-29T11:47:37.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.205+0000 7fa140c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:37.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.205+0000 7fa140c7b640 1 -- 172.21.15.55:0/889934629 >> v1:172.21.15.55:6789/0 conn(0x7fa13c13f650 legacy=0x7fa13c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:37.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.205+0000 7fa140c7b640 1 -- 172.21.15.55:0/889934629 shutdown_connections 2024-03-29T11:47:37.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.205+0000 7fa140c7b640 1 -- 172.21.15.55:0/889934629 >> 172.21.15.55:0/889934629 conn(0x7fa13c066a00 msgr2=0x7fa13c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:37.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.206+0000 7fa140c7b640 1 -- 172.21.15.55:0/889934629 shutdown_connections 2024-03-29T11:47:37.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.206+0000 7fa140c7b640 1 -- 172.21.15.55:0/889934629 wait complete. 2024-03-29T11:47:37.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.208+0000 7fa140c7b640 10 monclient: build_initial_monmap 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.208+0000 7fa140c7b640 10 monclient: monmap: 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.208+0000 7fa140c7b640 1 librados: starting msgr at 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.208+0000 7fa140c7b640 1 librados: starting objecter 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.208+0000 7fa140c7b640 1 Processor -- start 2024-03-29T11:47:37.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.208+0000 7fa140c7b640 1 -- start start 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.208+0000 7fa140c7b640 1 librados: setting wanted keys 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.208+0000 7fa140c7b640 1 librados: calling monclient init 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.208+0000 7fa140c7b640 10 monclient: init 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa13c0a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa13c13f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa13c1457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c131c40 con 0x7fa13c1457f0 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c06ebd0 con 0x7fa13c0a9d70 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c06eeb0 con 0x7fa13c13f650 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:37.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.209+0000 7fa140c7b640 10 monclient(hunting): authenticate will time out at 1270.488255s 2024-03-29T11:47:37.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.213+0000 7fa13ad9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa13c1457f0 0x7fa13c071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56701/0 (socket says 172.21.15.55:56701) 2024-03-29T11:47:37.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.213+0000 7fa13ad9d640 1 -- 172.21.15.55:0/1072856741 learned_addr learned my addr 172.21.15.55:0/1072856741 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:37.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.214+0000 7fa12affd640 10 client.?.objecter ms_handle_connect 0x7fa13c1457f0 2024-03-29T11:47:37.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.214+0000 7fa12affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:37.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.214+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812047680 0 0) 0x7fa130001270 con 0x7fa13c1457f0 2024-03-29T11:47:37.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.214+0000 7fa12affd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:37.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.214+0000 7fa12affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:37.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.214+0000 7fa12affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:37.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.214+0000 7fa12affd640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:47:37.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.214+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa11c004eb0 con 0x7fa13c1457f0 2024-03-29T11:47:37.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.217+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2929936950 0 0) 0x7fa11c004eb0 con 0x7fa13c1457f0 2024-03-29T11:47:37.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.217+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa130001270 con 0x7fa13c1457f0 2024-03-29T11:47:37.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.217+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa1300018a0 con 0x7fa13c1457f0 2024-03-29T11:47:37.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.217+0000 7fa12affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.217+0000 7fa12affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.217+0000 7fa12affd640 10 monclient(hunting): dump: 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.217+0000 7fa12affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.217+0000 7fa12affd640 10 client.?.objecter ms_handle_connect 0x7fa13c13f650 2024-03-29T11:47:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.217+0000 7fa12affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:37.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.218+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3459487310 0 0) 0x7fa130001270 con 0x7fa13c1457f0 2024-03-29T11:47:37.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.218+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 >> v1:172.21.15.55:6790/0 conn(0x7fa13c13f650 legacy=0x7fa13c135690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:37.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.218+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 >> v1:172.21.15.171:6789/0 conn(0x7fa13c0a9d70 legacy=0x7fa13c142880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:47:37.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.220+0000 7fa12affd640 10 monclient: _finish_hunting 0 2024-03-29T11:47:37.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.220+0000 7fa12affd640 1 monclient: found mon.a 2024-03-29T11:47:37.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.220+0000 7fa12affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:37.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.220+0000 7fa12affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:37.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.220+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c140db0 con 0x7fa13c1457f0 2024-03-29T11:47:37.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.220+0000 7fa12affd640 10 monclient: _finish_auth 0 2024-03-29T11:47:37.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.220+0000 7fa12affd640 10 monclient: _check_auth_tickets 2024-03-29T11:47:37.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.220+0000 7fa12affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:37.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.221+0000 7fa140c7b640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:47:37.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.221+0000 7fa140c7b640 10 monclient: _renew_subs 2024-03-29T11:47:37.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.221+0000 7fa140c7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:37.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.221+0000 7fa140c7b640 1 -- 172.21.15.55:0/1072856741 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13c141e50 con 0x7fa13c1457f0 2024-03-29T11:47:37.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.221+0000 7fa140c7b640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:37.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.221+0000 7fa140c7b640 10 monclient: _renew_subs 2024-03-29T11:47:37.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.221+0000 7fa140c7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:37.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.221+0000 7fa140c7b640 1 -- 172.21.15.55:0/1072856741 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c144a30 con 0x7fa13c1457f0 2024-03-29T11:47:37.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.221+0000 7fa140c7b640 1 librados: init done 2024-03-29T11:47:37.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.222+0000 7fa140c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:37.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.222+0000 7fa140c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:37.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.222+0000 7fa140c7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:37.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.222+0000 7fa140c7b640 1 -- 172.21.15.55:0/1072856741 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13c075f10 con 0x7fa13c1457f0 2024-03-29T11:47:37.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.222+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa130001a70 con 0x7fa13c1457f0 2024-03-29T11:47:37.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.222+0000 7fa12affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:37.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.222+0000 7fa138d99640 4 set_mon_vals no callback set 2024-03-29T11:47:37.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.224+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa1300038c0 con 0x7fa13c1457f0 2024-03-29T11:47:37.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.224+0000 7fa12affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:37.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.224+0000 7fa12affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.224+0000 7fa12affd640 10 monclient: dump: 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:37.230 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:37.231 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:37.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.229+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fa130003a20 con 0x7fa13c1457f0 2024-03-29T11:47:37.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.230+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fa130084a20 con 0x7fa13c1457f0 2024-03-29T11:47:37.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.230+0000 7fa12affd640 10 client.4425.objecter ms_dispatch 0x7fa13c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:47:37.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.230+0000 7fa12affd640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:47:37.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.230+0000 7fa12affd640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:47:37.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.230+0000 7fa12affd640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T11:47:37.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.230+0000 7fa13a59c640 10 monclient: get_auth_request con 0x7fa11c06dbf0 auth_method 0 2024-03-29T11:47:37.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.232+0000 7fa12affd640 10 client.4425.objecter ms_handle_connect 0x7fa11c06dbf0 2024-03-29T11:47:37.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.233+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa130050130 con 0x7fa13c1457f0 2024-03-29T11:47:37.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.233+0000 7fa12affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:37.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.233+0000 7fa12affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:37.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.564+0000 7fa140c7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:37.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.564+0000 7fa140c7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:37.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.564+0000 7fa140c7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:47:37.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.564+0000 7fa140c7b640 1 -- 172.21.15.55:0/1072856741 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa13c0b3680 con 0x7fa13c1457f0 2024-03-29T11:47:37.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.565+0000 7fa12affd640 1 -- 172.21.15.55:0/1072856741 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fa130052230 con 0x7fa13c1457f0 2024-03-29T11:47:37.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.565+0000 7fa12affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:37.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.565+0000 7fa12affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:37.570 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:47:37.570 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:37.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.568+0000 7fa128ff9640 10 librados: watch_flush enter 2024-03-29T11:47:37.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.568+0000 7fa128ff9640 10 librados: watch_flush exit 2024-03-29T11:47:37.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.568+0000 7fa128ff9640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T11:47:37.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.568+0000 7fa128ff9640 10 client.4425.objecter successfully canceled tick 2024-03-29T11:47:37.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.568+0000 7fa128ff9640 1 -- 172.21.15.55:0/1072856741 >> v1:172.21.15.171:6816/1463197163 conn(0x7fa11c06dbf0 legacy=0x7fa11c0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:37.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.568+0000 7fa128ff9640 10 monclient: shutdown 2024-03-29T11:47:37.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.568+0000 7fa128ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:37.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.568+0000 7fa128ff9640 1 -- 172.21.15.55:0/1072856741 >> v1:172.21.15.55:6789/0 conn(0x7fa13c1457f0 legacy=0x7fa13c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:37.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.568+0000 7fa128ff9640 1 -- 172.21.15.55:0/1072856741 shutdown_connections 2024-03-29T11:47:37.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.569+0000 7fa128ff9640 1 -- 172.21.15.55:0/1072856741 >> 172.21.15.55:0/1072856741 conn(0x7fa13c066a00 msgr2=0x7fa13c0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:37.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.569+0000 7fa128ff9640 1 -- 172.21.15.55:0/1072856741 shutdown_connections 2024-03-29T11:47:37.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.569+0000 7fa128ff9640 1 -- 172.21.15.55:0/1072856741 wait complete. 2024-03-29T11:47:37.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:37.569+0000 7fa128ff9640 1 librados: shutdown 2024-03-29T11:47:37.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:44.400 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:44.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.565+0000 7f559b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:47:44.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.565+0000 7f559b59e640 10 monclient: build_initial_monmap 2024-03-29T11:47:44.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.565+0000 7f559b59e640 10 monclient: monmap: 2024-03-29T11:47:44.572 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:44.572 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:44.572 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:47:44.566610+0000 2024-03-29T11:47:44.572 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:47:44.566610+0000 2024-03-29T11:47:44.572 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:44.573 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:44.573 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:44.573 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:47:44.573 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:47:44.573 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:44.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.566+0000 7f559b59e640 1 Processor -- start 2024-03-29T11:47:44.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.566+0000 7f559b59e640 1 -- start start 2024-03-29T11:47:44.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.566+0000 7f559b59e640 10 monclient: init 2024-03-29T11:47:44.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.567+0000 7f559b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:44.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.567+0000 7f559b59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:47:44.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.567+0000 7f559b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f559c0b2800 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:44.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.567+0000 7f559b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f559c0a9c00 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:44.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f559c0b38c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:44.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559b59e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c141eb0 con 0x7f559c0b38c0 2024-03-29T11:47:44.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559b59e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c0b22d0 con 0x7f559c0b2800 2024-03-29T11:47:44.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559a59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f559c0b2800 0x7f559c142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:52284/0 (socket says 172.21.15.55:52284) 2024-03-29T11:47:44.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559a59c640 1 -- 172.21.15.55:0/238701618 learned_addr learned my addr 172.21.15.55:0/238701618 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:44.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559b59e640 1 -- 172.21.15.55:0/238701618 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c142d30 con 0x7f559c0a9c00 2024-03-29T11:47:44.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:44.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559b59e640 10 monclient(hunting): authenticate will time out at 1277.848032s 2024-03-29T11:47:44.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559959a640 1 -- 172.21.15.55:0/238701618 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062539751 0 0) 0x7f559c0b22d0 con 0x7f559c0b2800 2024-03-29T11:47:44.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559959a640 10 monclient(hunting): _negotiate 2024-03-29T11:47:44.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:44.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:44.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559959a640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:47:44.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.568+0000 7f559959a640 1 -- 172.21.15.55:0/238701618 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5588003520 con 0x7f559c0b2800 2024-03-29T11:47:44.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 1 -- 172.21.15.55:0/238701618 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552012162 0 0) 0x7f5588003520 con 0x7f559c0b2800 2024-03-29T11:47:44.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 1 -- 172.21.15.55:0/238701618 >> v1:172.21.15.55:6790/0 conn(0x7f559c0a9c00 legacy=0x7f559c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:47:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 1 -- 172.21.15.55:0/238701618 >> v1:172.21.15.55:6789/0 conn(0x7f559c0b38c0 legacy=0x7f559c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: _finish_hunting 0 2024-03-29T11:47:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 1 monclient: found mon.noname-c 2024-03-29T11:47:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:47:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 1 -- 172.21.15.55:0/238701618 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c0aaf90 con 0x7f559c0b2800 2024-03-29T11:47:44.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: _finish_auth 0 2024-03-29T11:47:44.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: _check_auth_tickets 2024-03-29T11:47:44.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:44.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 1 -- 172.21.15.55:0/238701618 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5594001690 con 0x7f559c0b2800 2024-03-29T11:47:44.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:44.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:47:44.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: dump: 2024-03-29T11:47:44.587 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:44.587 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:44.587 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.587 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.587 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:44.587 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:44.588 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:44.588 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:44.588 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:44.588 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:44.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559b59e640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:47:44.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:44.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 1 -- 172.21.15.55:0/238701618 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55940018b0 con 0x7f559c0b2800 2024-03-29T11:47:44.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:44.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 1 -- 172.21.15.55:0/238701618 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5594001c40 con 0x7f559c0b2800 2024-03-29T11:47:44.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:44.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:47:44.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.569+0000 7f559959a640 10 monclient: dump: 2024-03-29T11:47:44.589 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:44.589 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:44.589 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.590 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.590 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:44.590 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:44.590 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:44.590 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:44.590 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:44.590 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:44.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.570+0000 7f559b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:44.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.570+0000 7f559b59e640 4 set_mon_vals no callback set 2024-03-29T11:47:44.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.570+0000 7f559b59e640 10 monclient: shutdown 2024-03-29T11:47:44.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.570+0000 7f559b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:44.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.570+0000 7f559b59e640 1 -- 172.21.15.55:0/238701618 >> v1:172.21.15.171:6789/0 conn(0x7f559c0b2800 legacy=0x7f559c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:44.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.571+0000 7f559b59e640 1 -- 172.21.15.55:0/238701618 shutdown_connections 2024-03-29T11:47:44.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.571+0000 7f559b59e640 1 -- 172.21.15.55:0/238701618 >> 172.21.15.55:0/238701618 conn(0x7f559c066a00 msgr2=0x7f559c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:44.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.571+0000 7f559b59e640 1 -- 172.21.15.55:0/238701618 shutdown_connections 2024-03-29T11:47:44.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.571+0000 7f559b59e640 1 -- 172.21.15.55:0/238701618 wait complete. 2024-03-29T11:47:44.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.572+0000 7f559b59e640 10 monclient: build_initial_monmap 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.572+0000 7f559b59e640 10 monclient: monmap: 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.573+0000 7f559b59e640 1 librados: starting msgr at 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.573+0000 7f559b59e640 1 librados: starting objecter 2024-03-29T11:47:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.573+0000 7f559b59e640 1 Processor -- start 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.573+0000 7f559b59e640 1 -- start start 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.573+0000 7f559b59e640 1 librados: setting wanted keys 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.573+0000 7f559b59e640 1 librados: calling monclient init 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.573+0000 7f559b59e640 10 monclient: init 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f559c0a9c00 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f559c0b2800 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f559c0b38c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c0ad180 con 0x7f559c0a9c00 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c138810 con 0x7f559c0b38c0 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c0abf30 con 0x7f559c0b2800 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:44.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559b59e640 10 monclient(hunting): authenticate will time out at 1277.854137s 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559a59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f559c0a9c00 0x7f559c1d7520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56452/0 (socket says 172.21.15.55:56452) 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f559a59c640 1 -- 172.21.15.55:0/952911109 learned_addr learned my addr 172.21.15.55:0/952911109 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f5592ffd640 10 client.?.objecter ms_handle_connect 0x7f559c0a9c00 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.574+0000 7f5592ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 client.?.objecter ms_handle_connect 0x7f559c0b38c0 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 client.?.objecter ms_handle_connect 0x7f559c0b2800 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835864760 0 0) 0x7f559c138810 con 0x7f559c0b38c0 2024-03-29T11:47:44.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f557c004eb0 con 0x7f559c0b38c0 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911321013 0 0) 0x7f559c0abf30 con 0x7f559c0b2800 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c138810 con 0x7f559c0b2800 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81057820 0 0) 0x7f559c0ad180 con 0x7f559c0a9c00 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:44.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:47:44.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.575+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c0abf30 con 0x7f559c0a9c00 2024-03-29T11:47:44.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3410768485 0 0) 0x7f557c004eb0 con 0x7f559c0b38c0 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f559c0ad180 con 0x7f559c0b38c0 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f55840017d0 con 0x7f559c0b38c0 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 10 monclient(hunting): dump: 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:44.599 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1071772040 0 0) 0x7f559c0abf30 con 0x7f559c0a9c00 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f557c004eb0 con 0x7f559c0a9c00 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5594001590 con 0x7f559c0a9c00 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.576+0000 7f5592ffd640 10 monclient(hunting): dump: 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:44.600 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1824999722 0 0) 0x7f559c138810 con 0x7f559c0b2800 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f559c0abf30 con 0x7f559c0b2800 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f558c001890 con 0x7f559c0b2800 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 10 monclient(hunting): dump: 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.601 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1223113824 0 0) 0x7f559c0ad180 con 0x7f559c0b38c0 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 >> v1:172.21.15.55:6790/0 conn(0x7f559c0b2800 legacy=0x7f559c1379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 >> v1:172.21.15.55:6789/0 conn(0x7f559c0a9c00 legacy=0x7f559c1d7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 1 monclient: found mon.b 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c1d7c30 con 0x7f559c0b38c0 2024-03-29T11:47:44.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 10 monclient: _finish_auth 0 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f5592ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f559b59e640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f559b59e640 10 monclient: _renew_subs 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f559b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c13c6e0 con 0x7f559c0b38c0 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f559b59e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f559b59e640 10 monclient: _renew_subs 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f559b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c13f3a0 con 0x7f559c0b38c0 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.577+0000 7f559b59e640 1 librados: init done 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.578+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5584001ce0 con 0x7f559c0b38c0 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.578+0000 7f5592ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.578+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5584003790 con 0x7f559c0b38c0 2024-03-29T11:47:44.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.578+0000 7f5592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:44.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.578+0000 7f5592ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:47:44.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.578+0000 7f5592ffd640 10 monclient: dump: 2024-03-29T11:47:44.604 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:44.604 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:44.604 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.606 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:44.606 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:44.606 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:44.606 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:44.606 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:44.606 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:44.606 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.578+0000 7f5598d99640 4 set_mon_vals no callback set 2024-03-29T11:47:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.579+0000 7f559b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.579+0000 7f559b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.579+0000 7f559b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.579+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559c04f730 con 0x7f559c0b38c0 2024-03-29T11:47:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.580+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f5584019760 con 0x7f559c0b38c0 2024-03-29T11:47:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.580+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f5584083730 con 0x7f559c0b38c0 2024-03-29T11:47:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.580+0000 7f5592ffd640 10 client.4426.objecter ms_dispatch 0x7f559c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:47:44.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.580+0000 7f5592ffd640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:47:44.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.580+0000 7f5592ffd640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:47:44.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.581+0000 7f559a59c640 10 monclient: get_auth_request con 0x7f557c06e140 auth_method 0 2024-03-29T11:47:44.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.581+0000 7f5592ffd640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T11:47:44.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.581+0000 7f5592ffd640 10 client.4426.objecter ms_handle_connect 0x7f557c06e140 2024-03-29T11:47:44.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.587+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f558404ee40 con 0x7f559c0b38c0 2024-03-29T11:47:44.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.587+0000 7f5592ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:44.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.587+0000 7f5592ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:44.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.980+0000 7f559b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:44.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.980+0000 7f559b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:44.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.980+0000 7f559b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:47:44.986 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.980+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f559c1d7dc0 con 0x7f559c0b38c0 2024-03-29T11:47:44.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.982+0000 7f5592ffd640 1 -- 172.21.15.55:0/952911109 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f5584050f40 con 0x7f559c0b38c0 2024-03-29T11:47:44.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.982+0000 7f5592ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:44.987 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.982+0000 7f5592ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:44.988 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:47:44.988 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:44.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.985+0000 7f559b59e640 10 librados: watch_flush enter 2024-03-29T11:47:44.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.985+0000 7f559b59e640 10 librados: watch_flush exit 2024-03-29T11:47:44.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.985+0000 7f559b59e640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T11:47:44.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.985+0000 7f559b59e640 10 client.4426.objecter successfully canceled tick 2024-03-29T11:47:44.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.985+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 >> v1:172.21.15.171:6816/1463197163 conn(0x7f557c06e140 legacy=0x7f557c070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:44.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.985+0000 7f559b59e640 10 monclient: shutdown 2024-03-29T11:47:44.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.985+0000 7f559b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:44.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.985+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 >> v1:172.21.15.171:6789/0 conn(0x7f559c0b38c0 legacy=0x7f559c138100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:44.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.986+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 shutdown_connections 2024-03-29T11:47:44.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.986+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 >> 172.21.15.55:0/952911109 conn(0x7f559c066a00 msgr2=0x7f559c140470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:44.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.986+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 shutdown_connections 2024-03-29T11:47:44.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.986+0000 7f559b59e640 1 -- 172.21.15.55:0/952911109 wait complete. 2024-03-29T11:47:44.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:44.986+0000 7f559b59e640 1 librados: shutdown 2024-03-29T11:47:45.005 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:51.813 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:51.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.958+0000 7fc218b32640 10 monclient: get_monmap_and_config 2024-03-29T11:47:51.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.958+0000 7fc218b32640 10 monclient: build_initial_monmap 2024-03-29T11:47:51.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.958+0000 7fc218b32640 10 monclient: monmap: 2024-03-29T11:47:51.964 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:51.965 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:51.965 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:47:51.960079+0000 2024-03-29T11:47:51.965 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:47:51.960079+0000 2024-03-29T11:47:51.965 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:51.965 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:51.965 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:51.965 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:47:51.965 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:47:51.965 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:51.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.959+0000 7fc218b32640 1 Processor -- start 2024-03-29T11:47:51.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.960+0000 7fc218b32640 1 -- start start 2024-03-29T11:47:51.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.960+0000 7fc218b32640 10 monclient: init 2024-03-29T11:47:51.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.961+0000 7fc218b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:51.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.961+0000 7fc218b32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:47:51.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.961+0000 7fc218b32640 10 monclient(hunting): picked mon.noname-c con 0x7fc20c0389e0 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:51.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.962+0000 7fc218b32640 10 monclient(hunting): picked mon.noname-b con 0x7fc20c038dc0 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:51.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.962+0000 7fc218b32640 10 monclient(hunting): picked mon.noname-a con 0x7fc20c030660 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:51.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.963+0000 7fc218b32640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c0c54c0 con 0x7fc20c030660 2024-03-29T11:47:51.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.963+0000 7fc218b32640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c036430 con 0x7fc20c0389e0 2024-03-29T11:47:51.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.963+0000 7fc218b32640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c039840 con 0x7fc20c038dc0 2024-03-29T11:47:51.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.963+0000 7fc218b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:51.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.963+0000 7fc218b32640 10 monclient(hunting): authenticate will time out at 1285.242680s 2024-03-29T11:47:51.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.964+0000 7fc212d9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc20c030660 0x7fc20c0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:36438/0 (socket says 172.21.15.55:36438) 2024-03-29T11:47:51.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.964+0000 7fc212d9d640 1 -- 172.21.15.55:0/676115624 learned_addr learned my addr 172.21.15.55:0/676115624 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:51.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.965+0000 7fc21159a640 1 -- 172.21.15.55:0/676115624 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895405493 0 0) 0x7fc20c0c54c0 con 0x7fc20c030660 2024-03-29T11:47:51.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.965+0000 7fc21159a640 10 monclient(hunting): _negotiate 2024-03-29T11:47:51.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.965+0000 7fc21159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:51.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.965+0000 7fc21159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:51.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.965+0000 7fc21159a640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:47:51.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.965+0000 7fc21159a640 1 -- 172.21.15.55:0/676115624 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1fc003520 con 0x7fc20c030660 2024-03-29T11:47:51.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.966+0000 7fc21159a640 1 -- 172.21.15.55:0/676115624 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 399847981 0 0) 0x7fc1fc003520 con 0x7fc20c030660 2024-03-29T11:47:51.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.966+0000 7fc21159a640 1 -- 172.21.15.55:0/676115624 >> v1:172.21.15.55:6790/0 conn(0x7fc20c038dc0 legacy=0x7fc20c0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:47:51.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.967+0000 7fc21159a640 1 -- 172.21.15.55:0/676115624 >> v1:172.21.15.171:6789/0 conn(0x7fc20c0389e0 legacy=0x7fc20c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.967+0000 7fc21159a640 10 monclient: _finish_hunting 0 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 1 monclient: found mon.noname-a 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 1 -- 172.21.15.55:0/676115624 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c035690 con 0x7fc20c030660 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 10 monclient: _finish_auth 0 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 10 monclient: _check_auth_tickets 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 1 -- 172.21.15.55:0/676115624 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc21404f730 con 0x7fc20c030660 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.968+0000 7fc21159a640 10 monclient: dump: 2024-03-29T11:47:51.977 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.970+0000 7fc218b32640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.970+0000 7fc218b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.971+0000 7fc21159a640 1 -- 172.21.15.55:0/676115624 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc214075f10 con 0x7fc20c030660 2024-03-29T11:47:51.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.971+0000 7fc21159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:51.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.971+0000 7fc21159a640 1 -- 172.21.15.55:0/676115624 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc2140758b0 con 0x7fc20c030660 2024-03-29T11:47:51.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.971+0000 7fc218b32640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:51.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.971+0000 7fc218b32640 4 set_mon_vals no callback set 2024-03-29T11:47:51.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.971+0000 7fc21159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:51.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.971+0000 7fc21159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:47:51.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.971+0000 7fc21159a640 10 monclient: dump: 2024-03-29T11:47:51.979 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:51.979 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:51.980 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:51.980 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:51.982 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:51.982 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:51.982 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:51.982 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:51.983 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:51.983 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.972+0000 7fc218b32640 10 monclient: shutdown 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.972+0000 7fc218b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.972+0000 7fc218b32640 1 -- 172.21.15.55:0/676115624 >> v1:172.21.15.55:6789/0 conn(0x7fc20c030660 legacy=0x7fc20c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.972+0000 7fc218b32640 1 -- 172.21.15.55:0/676115624 shutdown_connections 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.972+0000 7fc218b32640 1 -- 172.21.15.55:0/676115624 >> 172.21.15.55:0/676115624 conn(0x7fc20c005c00 msgr2=0x7fc20c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.972+0000 7fc218b32640 1 -- 172.21.15.55:0/676115624 shutdown_connections 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.972+0000 7fc218b32640 1 -- 172.21.15.55:0/676115624 wait complete. 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.974+0000 7fc218b32640 10 monclient: build_initial_monmap 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.974+0000 7fc218b32640 10 monclient: monmap: 2024-03-29T11:47:51.994 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.975+0000 7fc218b32640 1 librados: starting msgr at 2024-03-29T11:47:51.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.975+0000 7fc218b32640 1 librados: starting objecter 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.975+0000 7fc218b32640 1 Processor -- start 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.975+0000 7fc218b32640 1 -- start start 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.975+0000 7fc218b32640 1 librados: setting wanted keys 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.975+0000 7fc218b32640 1 librados: calling monclient init 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.975+0000 7fc218b32640 10 monclient: init 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.976+0000 7fc218b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.977+0000 7fc218b32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.978+0000 7fc218b32640 10 monclient(hunting): picked mon.noname-c con 0x7fc20c030660 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.978+0000 7fc218b32640 10 monclient(hunting): picked mon.noname-b con 0x7fc20c0389e0 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.978+0000 7fc218b32640 10 monclient(hunting): picked mon.noname-a con 0x7fc20c0c83b0 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:51.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.978+0000 7fc218b32640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c15fd20 con 0x7fc20c0c83b0 2024-03-29T11:47:51.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.978+0000 7fc218b32640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c160950 con 0x7fc20c0389e0 2024-03-29T11:47:51.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.978+0000 7fc218b32640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c160ae0 con 0x7fc20c030660 2024-03-29T11:47:51.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.978+0000 7fc218b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:51.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.978+0000 7fc218b32640 10 monclient(hunting): authenticate will time out at 1285.257811s 2024-03-29T11:47:51.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc212d9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc20c0c83b0 0x7fc20c0c0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41020/0 (socket says 172.21.15.55:41020) 2024-03-29T11:47:51.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc212d9d640 1 -- 172.21.15.55:0/291175873 learned_addr learned my addr 172.21.15.55:0/291175873 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:51.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 10 client.?.objecter ms_handle_connect 0x7fc20c0c83b0 2024-03-29T11:47:51.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:51.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369570886 0 0) 0x7fc20c15fd20 con 0x7fc20c0c83b0 2024-03-29T11:47:51.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:51.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:51.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:51.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:47:51.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f4004eb0 con 0x7fc20c0c83b0 2024-03-29T11:47:51.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 10 client.?.objecter ms_handle_connect 0x7fc20c030660 2024-03-29T11:47:51.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.979+0000 7fc202ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:51.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701751089 0 0) 0x7fc204001270 con 0x7fc20c030660 2024-03-29T11:47:51.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc20c15fd20 con 0x7fc20c030660 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172318115 0 0) 0x7fc20c15fd20 con 0x7fc20c030660 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc204001270 con 0x7fc20c030660 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc204001960 con 0x7fc20c030660 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 10 monclient(hunting): dump: 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:51.999 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.980+0000 7fc202ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 577915326 0 0) 0x7fc1f4004eb0 con 0x7fc20c0c83b0 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc20c15fd20 con 0x7fc20c0c83b0 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc214068810 con 0x7fc20c0c83b0 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:47:52.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 10 monclient(hunting): dump: 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1516550667 0 0) 0x7fc204001270 con 0x7fc20c030660 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 >> v1:172.21.15.171:6789/0 conn(0x7fc20c0389e0 legacy=0x7fc20c02d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 >> v1:172.21.15.55:6789/0 conn(0x7fc20c0c83b0 legacy=0x7fc20c0c0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:52.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 1 monclient: found mon.c 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c02dad0 con 0x7fc20c030660 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 10 monclient: _finish_auth 0 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc202ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc218b32640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc218b32640 10 monclient: _renew_subs 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc218b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc218b32640 1 -- 172.21.15.55:0/291175873 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20c02f0b0 con 0x7fc20c030660 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc218b32640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:52.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc218b32640 10 monclient: _renew_subs 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc218b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc218b32640 1 -- 172.21.15.55:0/291175873 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20c0c08a0 con 0x7fc20c030660 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.983+0000 7fc218b32640 1 librados: init done 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc200ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc200ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc200ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc204003610 con 0x7fc20c030660 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc200ff9640 1 -- 172.21.15.55:0/291175873 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20c02dc80 con 0x7fc20c030660 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc202ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc204003920 con 0x7fc20c030660 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc202ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc202ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.985+0000 7fc202ffd640 10 monclient: dump: 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:52.003 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.986+0000 7fc203fff640 4 set_mon_vals no callback set 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.986+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fc2040199b0 con 0x7fc20c030660 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.986+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fc2040839a0 con 0x7fc20c030660 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.986+0000 7fc202ffd640 10 client.4445.objecter ms_dispatch 0x7fc214000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.986+0000 7fc202ffd640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.986+0000 7fc202ffd640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:47:52.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.986+0000 7fc202ffd640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:47:52.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.987+0000 7fc211d9b640 10 monclient: get_auth_request con 0x7fc1f406dd50 auth_method 0 2024-03-29T11:47:52.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.988+0000 7fc202ffd640 10 client.4445.objecter ms_handle_connect 0x7fc1f406dd50 2024-03-29T11:47:52.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.993+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc20404f170 con 0x7fc20c030660 2024-03-29T11:47:52.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.993+0000 7fc202ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:52.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:51.993+0000 7fc202ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:52.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.324+0000 7fc200ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:52.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.324+0000 7fc200ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:52.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.324+0000 7fc200ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:52.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.324+0000 7fc200ff9640 1 -- 172.21.15.55:0/291175873 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc20c007b70 con 0x7fc20c030660 2024-03-29T11:47:52.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.328+0000 7fc202ffd640 1 -- 172.21.15.55:0/291175873 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fc204051270 con 0x7fc20c030660 2024-03-29T11:47:52.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.328+0000 7fc202ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:52.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.328+0000 7fc202ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:52.333 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:47:52.334 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:52.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.331+0000 7fc200ff9640 10 librados: watch_flush enter 2024-03-29T11:47:52.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.332+0000 7fc200ff9640 10 librados: watch_flush exit 2024-03-29T11:47:52.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.332+0000 7fc200ff9640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:47:52.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.332+0000 7fc200ff9640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:47:52.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.332+0000 7fc200ff9640 1 -- 172.21.15.55:0/291175873 >> v1:172.21.15.171:6816/1463197163 conn(0x7fc1f406dd50 legacy=0x7fc1f4070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:52.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.332+0000 7fc200ff9640 10 monclient: shutdown 2024-03-29T11:47:52.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.332+0000 7fc200ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:52.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.332+0000 7fc200ff9640 1 -- 172.21.15.55:0/291175873 >> v1:172.21.15.55:6790/0 conn(0x7fc20c030660 legacy=0x7fc20c160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:52.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.332+0000 7fc200ff9640 1 -- 172.21.15.55:0/291175873 shutdown_connections 2024-03-29T11:47:52.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.332+0000 7fc200ff9640 1 -- 172.21.15.55:0/291175873 >> 172.21.15.55:0/291175873 conn(0x7fc20c005c00 msgr2=0x7fc20c005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:52.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.333+0000 7fc200ff9640 1 -- 172.21.15.55:0/291175873 shutdown_connections 2024-03-29T11:47:52.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.333+0000 7fc200ff9640 1 -- 172.21.15.55:0/291175873 wait complete. 2024-03-29T11:47:52.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:52.333+0000 7fc200ff9640 1 librados: shutdown 2024-03-29T11:47:52.356 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:59.163 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.293+0000 7f8a1f558640 10 monclient: get_monmap_and_config 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.293+0000 7f8a1f558640 10 monclient: build_initial_monmap 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.293+0000 7f8a1f558640 10 monclient: monmap: 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:47:59.294425+0000 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:47:59.294425+0000 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:59.299 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:59.300 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:47:59.300 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:47:59.300 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:59.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.294+0000 7f8a1f558640 1 Processor -- start 2024-03-29T11:47:59.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.294+0000 7f8a1f558640 1 -- start start 2024-03-29T11:47:59.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.294+0000 7f8a1f558640 10 monclient: init 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 10 monclient(hunting): picked mon.noname-c con 0x7f8a181457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 10 monclient(hunting): picked mon.noname-b con 0x7f8a1813f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 10 monclient(hunting): picked mon.noname-a con 0x7f8a180a9d70 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180aaef0 con 0x7f8a180a9d70 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a18145bd0 con 0x7f8a181457f0 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a18145eb0 con 0x7f8a1813f650 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.295+0000 7f8a1f558640 10 monclient(hunting): authenticate will time out at 1292.574862s 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a1caf3640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f8a1813f650 0x7f8a180b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:37380/0 (socket says 172.21.15.55:37380) 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a1caf3640 1 -- 172.21.15.55:0/423300330 learned_addr learned my addr 172.21.15.55:0/423300330 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a0ffff640 1 -- 172.21.15.55:0/423300330 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229351551 0 0) 0x7f8a18145eb0 con 0x7f8a1813f650 2024-03-29T11:47:59.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:47:59.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:59.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:59.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a0ffff640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:47:59.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a0ffff640 1 -- 172.21.15.55:0/423300330 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a04003520 con 0x7f8a1813f650 2024-03-29T11:47:59.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a0ffff640 1 -- 172.21.15.55:0/423300330 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2953566447 0 0) 0x7f8a04003520 con 0x7f8a1813f650 2024-03-29T11:47:59.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a0ffff640 1 -- 172.21.15.55:0/423300330 >> v1:172.21.15.171:6789/0 conn(0x7f8a181457f0 legacy=0x7f8a1813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:47:59.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.296+0000 7f8a0ffff640 1 -- 172.21.15.55:0/423300330 >> v1:172.21.15.55:6789/0 conn(0x7f8a180a9d70 legacy=0x7f8a1814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:47:59.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:47:59.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 1 monclient: found mon.noname-b 2024-03-29T11:47:59.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:59.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:47:59.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 1 -- 172.21.15.55:0/423300330 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a18146340 con 0x7f8a1813f650 2024-03-29T11:47:59.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: _finish_auth 0 2024-03-29T11:47:59.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:47:59.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:59.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a1f558640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:47:59.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a1f558640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:59.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 1 -- 172.21.15.55:0/423300330 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8a10001a10 con 0x7f8a1813f650 2024-03-29T11:47:59.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:59.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:59.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: dump: 2024-03-29T11:47:59.311 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a1f558640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 1 -- 172.21.15.55:0/423300330 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a10001ca0 con 0x7f8a1813f650 2024-03-29T11:47:59.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 1 -- 172.21.15.55:0/423300330 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8a10002030 con 0x7f8a1813f650 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a1f558640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a1f558640 4 set_mon_vals no callback set 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.298+0000 7f8a0ffff640 10 monclient: dump: 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:59.313 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.299+0000 7f8a1f558640 10 monclient: shutdown 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.299+0000 7f8a1f558640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.299+0000 7f8a1f558640 1 -- 172.21.15.55:0/423300330 >> v1:172.21.15.55:6790/0 conn(0x7f8a1813f650 legacy=0x7f8a180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.299+0000 7f8a1f558640 1 -- 172.21.15.55:0/423300330 shutdown_connections 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.299+0000 7f8a1f558640 1 -- 172.21.15.55:0/423300330 >> 172.21.15.55:0/423300330 conn(0x7f8a18066a00 msgr2=0x7f8a18066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.299+0000 7f8a1f558640 1 -- 172.21.15.55:0/423300330 shutdown_connections 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.299+0000 7f8a1f558640 1 -- 172.21.15.55:0/423300330 wait complete. 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.300+0000 7f8a1f558640 10 monclient: build_initial_monmap 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.300+0000 7f8a1f558640 10 monclient: monmap: 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:59.314 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 1 librados: starting msgr at 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 1 librados: starting objecter 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 1 Processor -- start 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 1 -- start start 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 1 librados: setting wanted keys 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 1 librados: calling monclient init 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 10 monclient: init 2024-03-29T11:47:59.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.301+0000 7f8a1f558640 10 monclient(hunting): picked mon.noname-c con 0x7f8a180a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a1f558640 10 monclient(hunting): picked mon.noname-a con 0x7f8a1813f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a1f558640 10 monclient(hunting): picked mon.noname-b con 0x7f8a181457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a1f558640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1813b970 con 0x7f8a1813f650 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a1f558640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a18139af0 con 0x7f8a181457f0 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a1f558640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a18139c80 con 0x7f8a180a9d70 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a1f558640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a1f558640 10 monclient(hunting): authenticate will time out at 1292.581687s 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a1d2f4640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f8a180a9d70 0x7f8a1813b110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:38184/0 (socket says 172.21.15.55:38184) 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a1d2f4640 1 -- 172.21.15.55:0/1554140885 learned_addr learned my addr 172.21.15.55:0/1554140885 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 10 client.?.objecter ms_handle_connect 0x7f8a180a9d70 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939902661 0 0) 0x7f8a18139c80 con 0x7f8a180a9d70 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:47:59.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc004eb0 con 0x7f8a180a9d70 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415403731 0 0) 0x7f89fc004eb0 con 0x7f8a180a9d70 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a18139c80 con 0x7f8a180a9d70 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8a08003fe0 con 0x7f8a180a9d70 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.302+0000 7f8a0dffb640 10 monclient(hunting): dump: 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:59.317 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:59.318 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:59.318 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:59.318 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:59.318 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:59.318 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:59.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.303+0000 7f8a0dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:59.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.303+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2897630468 0 0) 0x7f8a18139c80 con 0x7f8a180a9d70 2024-03-29T11:47:59.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.303+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 >> v1:172.21.15.171:6789/0 conn(0x7f8a181457f0 legacy=0x7f8a180b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:59.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.303+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 >> v1:172.21.15.55:6789/0 conn(0x7f8a1813f650 legacy=0x7f8a181393e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:47:59.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.305+0000 7f8a0dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:47:59.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.305+0000 7f8a0dffb640 1 monclient: found mon.c 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.305+0000 7f8a0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.305+0000 7f8a0dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.305+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a18138690 con 0x7f8a180a9d70 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0dffb640 10 monclient: _finish_auth 0 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a08000ed0 con 0x7f8a180a9d70 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a1f558640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a1f558640 10 monclient: _renew_subs 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a1f558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:59.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a18138940 con 0x7f8a180a9d70 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8a080012a0 con 0x7f8a180a9d70 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0dffb640 10 monclient: dump: 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:47:59.323 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.306+0000 7f8a0effd640 4 set_mon_vals no callback set 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.307+0000 7f8a1f558640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.307+0000 7f8a1f558640 10 monclient: _renew_subs 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.307+0000 7f8a1f558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.307+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a18142540 con 0x7f8a180a9d70 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.307+0000 7f8a1f558640 1 librados: init done 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a1f558640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a1f558640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:59.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a1f558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:59.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a18142890 con 0x7f8a180a9d70 2024-03-29T11:47:59.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f8a08001400 con 0x7f8a180a9d70 2024-03-29T11:47:59.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f8a08084aa0 con 0x7f8a180a9d70 2024-03-29T11:47:59.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a0dffb640 10 client.4451.objecter ms_dispatch 0x7f8a18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:47:59.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a0dffb640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:47:59.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a0dffb640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:47:59.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.309+0000 7f8a0dffb640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:47:59.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.310+0000 7f8a1caf3640 10 monclient: get_auth_request con 0x7f89fc06db30 auth_method 0 2024-03-29T11:47:59.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.311+0000 7f8a0dffb640 10 client.4451.objecter ms_handle_connect 0x7f89fc06db30 2024-03-29T11:47:59.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.316+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a080503e0 con 0x7f8a180a9d70 2024-03-29T11:47:59.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.316+0000 7f8a0dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:59.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.316+0000 7f8a0dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:59.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.643+0000 7f8a1f558640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:59.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.643+0000 7f8a1f558640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:59.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.643+0000 7f8a1f558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:47:59.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.643+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a18075f10 con 0x7f8a180a9d70 2024-03-29T11:47:59.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.645+0000 7f8a0dffb640 1 -- 172.21.15.55:0/1554140885 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f8a08084150 con 0x7f8a180a9d70 2024-03-29T11:47:59.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.645+0000 7f8a0dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:59.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.645+0000 7f8a0dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:59.651 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:47:59.651 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.648+0000 7f8a1f558640 10 librados: watch_flush enter 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.648+0000 7f8a1f558640 10 librados: watch_flush exit 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.648+0000 7f8a1f558640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.648+0000 7f8a1f558640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.649+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 >> v1:172.21.15.171:6816/1463197163 conn(0x7f89fc06db30 legacy=0x7f89fc06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.649+0000 7f8a1f558640 10 monclient: shutdown 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.649+0000 7f8a1f558640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.649+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 >> v1:172.21.15.55:6790/0 conn(0x7f8a180a9d70 legacy=0x7f8a1813b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.649+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 shutdown_connections 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.649+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 >> 172.21.15.55:0/1554140885 conn(0x7f8a18066a00 msgr2=0x7f8a180b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.649+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 shutdown_connections 2024-03-29T11:47:59.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.649+0000 7f8a1f558640 1 -- 172.21.15.55:0/1554140885 wait complete. 2024-03-29T11:47:59.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:47:59.649+0000 7f8a1f558640 1 librados: shutdown 2024-03-29T11:47:59.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:06.475 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.626+0000 7fc18ef15640 10 monclient: get_monmap_and_config 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.626+0000 7fc18ef15640 10 monclient: build_initial_monmap 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.626+0000 7fc18ef15640 10 monclient: monmap: 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:48:06.627970+0000 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:48:06.627970+0000 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:06.633 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:48:06.634 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:48:06.634 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:06.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.628+0000 7fc18ef15640 1 Processor -- start 2024-03-29T11:48:06.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.628+0000 7fc18ef15640 1 -- start start 2024-03-29T11:48:06.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.628+0000 7fc18ef15640 10 monclient: init 2024-03-29T11:48:06.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:06.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:06.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 10 monclient(hunting): picked mon.noname-b con 0x7fc1880ae250 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:06.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 10 monclient(hunting): picked mon.noname-c con 0x7fc18813f970 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:06.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 10 monclient(hunting): picked mon.noname-a con 0x7fc188142800 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:06.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1880afe00 con 0x7fc188142800 2024-03-29T11:48:06.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc188142c10 con 0x7fc18813f970 2024-03-29T11:48:06.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1880afb20 con 0x7fc1880ae250 2024-03-29T11:48:06.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:06.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.629+0000 7fc18ef15640 10 monclient(hunting): authenticate will time out at 1299.909248s 2024-03-29T11:48:06.639 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.632+0000 7fc18d4b2640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc188142800 0x7fc18814da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:55425/0 (socket says 172.21.15.55:55425) 2024-03-29T11:48:06.639 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.632+0000 7fc18d4b2640 1 -- 172.21.15.55:0/1603591296 learned_addr learned my addr 172.21.15.55:0/1603591296 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:06.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.633+0000 7fc1877fe640 1 -- 172.21.15.55:0/1603591296 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094454712 0 0) 0x7fc1880afe00 con 0x7fc188142800 2024-03-29T11:48:06.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.633+0000 7fc1877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:06.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.633+0000 7fc1877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:06.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.633+0000 7fc1877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:06.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.633+0000 7fc1877fe640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:48:06.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.633+0000 7fc1877fe640 1 -- 172.21.15.55:0/1603591296 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc170003520 con 0x7fc188142800 2024-03-29T11:48:06.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.633+0000 7fc1877fe640 1 -- 172.21.15.55:0/1603591296 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2747653970 0 0) 0x7fc170003520 con 0x7fc188142800 2024-03-29T11:48:06.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.633+0000 7fc1877fe640 1 -- 172.21.15.55:0/1603591296 >> v1:172.21.15.55:6790/0 conn(0x7fc1880ae250 legacy=0x7fc18813f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:48:06.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 1 -- 172.21.15.55:0/1603591296 >> v1:172.21.15.171:6789/0 conn(0x7fc18813f970 legacy=0x7fc1881420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:06.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 10 monclient: _finish_hunting 0 2024-03-29T11:48:06.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 1 monclient: found mon.noname-a 2024-03-29T11:48:06.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:06.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:48:06.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 1 -- 172.21.15.55:0/1603591296 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1880aab80 con 0x7fc188142800 2024-03-29T11:48:06.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 10 monclient: _finish_auth 0 2024-03-29T11:48:06.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 10 monclient: _check_auth_tickets 2024-03-29T11:48:06.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 1 -- 172.21.15.55:0/1603591296 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc178001690 con 0x7fc188142800 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.636+0000 7fc1877fe640 10 monclient: dump: 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:06.645 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:06.646 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:06.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.637+0000 7fc18ef15640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:48:06.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.637+0000 7fc18ef15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:06.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.638+0000 7fc1877fe640 1 -- 172.21.15.55:0/1603591296 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc178001860 con 0x7fc188142800 2024-03-29T11:48:06.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.638+0000 7fc1877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:06.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.638+0000 7fc1877fe640 1 -- 172.21.15.55:0/1603591296 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc178001cb0 con 0x7fc188142800 2024-03-29T11:48:06.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.638+0000 7fc1877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:06.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.638+0000 7fc1877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:06.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.638+0000 7fc1877fe640 10 monclient: dump: 2024-03-29T11:48:06.648 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:06.648 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:06.648 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.640+0000 7fc18ef15640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.640+0000 7fc18ef15640 4 set_mon_vals no callback set 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.641+0000 7fc18ef15640 10 monclient: shutdown 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.641+0000 7fc18ef15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:06.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.641+0000 7fc18ef15640 1 -- 172.21.15.55:0/1603591296 >> v1:172.21.15.55:6789/0 conn(0x7fc188142800 legacy=0x7fc18814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:06.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.642+0000 7fc18ef15640 1 -- 172.21.15.55:0/1603591296 shutdown_connections 2024-03-29T11:48:06.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.642+0000 7fc18ef15640 1 -- 172.21.15.55:0/1603591296 >> 172.21.15.55:0/1603591296 conn(0x7fc1880b35e0 msgr2=0x7fc1880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:06.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.642+0000 7fc18ef15640 1 -- 172.21.15.55:0/1603591296 shutdown_connections 2024-03-29T11:48:06.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.643+0000 7fc18ef15640 1 -- 172.21.15.55:0/1603591296 wait complete. 2024-03-29T11:48:06.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.644+0000 7fc18ef15640 10 monclient: build_initial_monmap 2024-03-29T11:48:06.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.644+0000 7fc18ef15640 10 monclient: monmap: 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:48:06.652 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:06.653 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.644+0000 7fc18ef15640 1 librados: starting msgr at 2024-03-29T11:48:06.653 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.644+0000 7fc18ef15640 1 librados: starting objecter 2024-03-29T11:48:06.653 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.644+0000 7fc18ef15640 1 Processor -- start 2024-03-29T11:48:06.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.645+0000 7fc18ef15640 1 -- start start 2024-03-29T11:48:06.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.645+0000 7fc18ef15640 1 librados: setting wanted keys 2024-03-29T11:48:06.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.645+0000 7fc18ef15640 1 librados: calling monclient init 2024-03-29T11:48:06.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.645+0000 7fc18ef15640 10 monclient: init 2024-03-29T11:48:06.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.645+0000 7fc18ef15640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:06.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.645+0000 7fc18ef15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:48:06.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.648+0000 7fc18ef15640 10 monclient(hunting): picked mon.noname-a con 0x7fc1880ae250 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:06.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.648+0000 7fc18ef15640 10 monclient(hunting): picked mon.noname-c con 0x7fc18813f970 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:06.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.648+0000 7fc18ef15640 10 monclient(hunting): picked mon.noname-b con 0x7fc188142800 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:06.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.648+0000 7fc18ef15640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1881db190 con 0x7fc1880ae250 2024-03-29T11:48:06.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.648+0000 7fc18ef15640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1881464d0 con 0x7fc188142800 2024-03-29T11:48:06.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.648+0000 7fc18ef15640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1881dac70 con 0x7fc18813f970 2024-03-29T11:48:06.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.648+0000 7fc18ef15640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:06.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.648+0000 7fc18ef15640 10 monclient(hunting): authenticate will time out at 1299.928228s 2024-03-29T11:48:06.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc18d4b2640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc188142800 0x7fc1881e2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:37831/0 (socket says 172.21.15.55:37831) 2024-03-29T11:48:06.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc18d4b2640 1 -- 172.21.15.55:0/3423075291 learned_addr learned my addr 172.21.15.55:0/3423075291 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:06.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 client.?.objecter ms_handle_connect 0x7fc18813f970 2024-03-29T11:48:06.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:06.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 client.?.objecter ms_handle_connect 0x7fc188142800 2024-03-29T11:48:06.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:06.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805851491 0 0) 0x7fc1881dac70 con 0x7fc18813f970 2024-03-29T11:48:06.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:06.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:06.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:06.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:48:06.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc174004eb0 con 0x7fc18813f970 2024-03-29T11:48:06.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982023963 0 0) 0x7fc1881464d0 con 0x7fc188142800 2024-03-29T11:48:06.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:06.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:06.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:06.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:48:06.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.651+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1881dac70 con 0x7fc188142800 2024-03-29T11:48:06.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1039306332 0 0) 0x7fc174004eb0 con 0x7fc18813f970 2024-03-29T11:48:06.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1881464d0 con 0x7fc18813f970 2024-03-29T11:48:06.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc180001c40 con 0x7fc18813f970 2024-03-29T11:48:06.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:06.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:06.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 10 monclient(hunting): dump: 2024-03-29T11:48:06.669 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:06.669 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:06.669 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.670 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.670 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:06.670 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:06.670 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:06.670 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:06.670 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:06.670 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:06.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2338014648 0 0) 0x7fc1881dac70 con 0x7fc188142800 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc174004eb0 con 0x7fc188142800 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc178001b70 con 0x7fc188142800 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.652+0000 7fc1857fa640 10 monclient(hunting): dump: 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.671 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:06.672 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:06.672 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:06.672 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:06.672 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:06.672 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:06.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1715141673 0 0) 0x7fc174004eb0 con 0x7fc188142800 2024-03-29T11:48:06.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 >> v1:172.21.15.55:6790/0 conn(0x7fc18813f970 legacy=0x7fc1881d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:06.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 >> v1:172.21.15.55:6789/0 conn(0x7fc1880ae250 legacy=0x7fc1881db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:06.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 10 monclient: _finish_hunting 0 2024-03-29T11:48:06.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 1 monclient: found mon.b 2024-03-29T11:48:06.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:06.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:06.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1881dae00 con 0x7fc188142800 2024-03-29T11:48:06.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 10 monclient: _finish_auth 0 2024-03-29T11:48:06.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 10 monclient: _check_auth_tickets 2024-03-29T11:48:06.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.654+0000 7fc1857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:06.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.655+0000 7fc18ef15640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:48:06.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.655+0000 7fc18ef15640 10 monclient: _renew_subs 2024-03-29T11:48:06.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.655+0000 7fc18ef15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:06.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.655+0000 7fc18ef15640 1 -- 172.21.15.55:0/3423075291 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc188145a30 con 0x7fc188142800 2024-03-29T11:48:06.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.655+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc178004a90 con 0x7fc188142800 2024-03-29T11:48:06.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.655+0000 7fc1857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:06.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.655+0000 7fc186ffd640 4 set_mon_vals no callback set 2024-03-29T11:48:06.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.656+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fc178004da0 con 0x7fc188142800 2024-03-29T11:48:06.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.656+0000 7fc1857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:06.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.656+0000 7fc1857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:06.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.656+0000 7fc1857fa640 10 monclient: dump: 2024-03-29T11:48:06.679 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:06.679 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:06.679 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.679 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:06.679 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:06.679 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.656+0000 7fc18ef15640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.656+0000 7fc18ef15640 10 monclient: _renew_subs 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.656+0000 7fc18ef15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.656+0000 7fc18ef15640 1 -- 172.21.15.55:0/3423075291 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1880acbb0 con 0x7fc188142800 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.656+0000 7fc18ef15640 1 librados: init done 2024-03-29T11:48:06.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.658+0000 7fc16effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:06.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.658+0000 7fc16effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:06.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.658+0000 7fc16effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:06.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.658+0000 7fc16effd640 1 -- 172.21.15.55:0/3423075291 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1881442b0 con 0x7fc188142800 2024-03-29T11:48:06.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.662+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fc17801ad70 con 0x7fc188142800 2024-03-29T11:48:06.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.662+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fc178084a20 con 0x7fc188142800 2024-03-29T11:48:06.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.662+0000 7fc1857fa640 10 client.4435.objecter ms_dispatch 0x7fc188000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:06.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.662+0000 7fc1857fa640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:06.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.662+0000 7fc1857fa640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:06.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.662+0000 7fc1857fa640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-29T11:48:06.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.663+0000 7fc18ccb1640 10 monclient: get_auth_request con 0x7fc17406de30 auth_method 0 2024-03-29T11:48:06.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.663+0000 7fc1857fa640 10 client.4435.objecter ms_handle_connect 0x7fc17406de30 2024-03-29T11:48:06.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.669+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc178050300 con 0x7fc188142800 2024-03-29T11:48:06.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.669+0000 7fc1857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:06.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:06.669+0000 7fc1857fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:07.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.006+0000 7fc16effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:07.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.006+0000 7fc16effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:07.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.006+0000 7fc16effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:07.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.007+0000 7fc16effd640 1 -- 172.21.15.55:0/3423075291 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1881db340 con 0x7fc188142800 2024-03-29T11:48:07.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.008+0000 7fc1857fa640 1 -- 172.21.15.55:0/3423075291 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fc178052230 con 0x7fc188142800 2024-03-29T11:48:07.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.008+0000 7fc1857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:07.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.008+0000 7fc1857fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:07.016 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:48:07.016 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:07.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.013+0000 7fc16effd640 10 librados: watch_flush enter 2024-03-29T11:48:07.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.015+0000 7fc16effd640 10 librados: watch_flush exit 2024-03-29T11:48:07.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.015+0000 7fc16effd640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-29T11:48:07.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.015+0000 7fc16effd640 10 client.4435.objecter successfully canceled tick 2024-03-29T11:48:07.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.015+0000 7fc16effd640 1 -- 172.21.15.55:0/3423075291 >> v1:172.21.15.171:6816/1463197163 conn(0x7fc17406de30 legacy=0x7fc1740702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:07.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.015+0000 7fc16effd640 10 monclient: shutdown 2024-03-29T11:48:07.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.015+0000 7fc16effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:07.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.015+0000 7fc16effd640 1 -- 172.21.15.55:0/3423075291 >> v1:172.21.15.171:6789/0 conn(0x7fc188142800 legacy=0x7fc1881e2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:07.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.015+0000 7fc16effd640 1 -- 172.21.15.55:0/3423075291 shutdown_connections 2024-03-29T11:48:07.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.015+0000 7fc16effd640 1 -- 172.21.15.55:0/3423075291 >> 172.21.15.55:0/3423075291 conn(0x7fc1880b35e0 msgr2=0x7fc1880b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:07.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.016+0000 7fc16effd640 1 -- 172.21.15.55:0/3423075291 shutdown_connections 2024-03-29T11:48:07.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.017+0000 7fc16effd640 1 -- 172.21.15.55:0/3423075291 wait complete. 2024-03-29T11:48:07.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:07.018+0000 7fc16effd640 1 librados: shutdown 2024-03-29T11:48:07.044 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:13.852 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:14.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.003+0000 7f3c5a219640 10 monclient: get_monmap_and_config 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.003+0000 7f3c5a219640 10 monclient: build_initial_monmap 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.004+0000 7f3c5a219640 10 monclient: monmap: 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:48:14.005138+0000 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:48:14.005138+0000 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:48:14.011 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:14.012 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.006+0000 7f3c5a219640 1 Processor -- start 2024-03-29T11:48:14.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.007+0000 7f3c5a219640 1 -- start start 2024-03-29T11:48:14.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.007+0000 7f3c5a219640 10 monclient: init 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 10 monclient(hunting): picked mon.noname-a con 0x7f3c540aaee0 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 10 monclient(hunting): picked mon.noname-b con 0x7f3c540aa500 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 10 monclient(hunting): picked mon.noname-c con 0x7f3c54149800 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c540aded0 con 0x7f3c540aaee0 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c540ad7e0 con 0x7f3c54149800 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c540ac350 con 0x7f3c540aa500 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:14.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.008+0000 7f3c5a219640 10 monclient(hunting): authenticate will time out at 1307.287949s 2024-03-29T11:48:14.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.011+0000 7f3c59a18640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3c54149800 0x7f3c54151ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:36985/0 (socket says 172.21.15.55:36985) 2024-03-29T11:48:14.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.011+0000 7f3c59a18640 1 -- 172.21.15.55:0/3535779347 learned_addr learned my addr 172.21.15.55:0/3535779347 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:14.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321285455 0 0) 0x7f3c540ad7e0 con 0x7f3c54149800 2024-03-29T11:48:14.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:48:14.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:14.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:14.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:48:14.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c40003520 con 0x7f3c54149800 2024-03-29T11:48:14.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291865889 0 0) 0x7f3c540aded0 con 0x7f3c540aaee0 2024-03-29T11:48:14.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:48:14.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:14.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:14.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:48:14.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c540ad7e0 con 0x7f3c540aaee0 2024-03-29T11:48:14.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1902803848 0 0) 0x7f3c40003520 con 0x7f3c54149800 2024-03-29T11:48:14.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.012+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 >> v1:172.21.15.55:6790/0 conn(0x7f3c540aa500 legacy=0x7f3c540ad0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:48:14.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 >> v1:172.21.15.55:6789/0 conn(0x7f3c540aaee0 legacy=0x7f3c5414a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:14.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:48:14.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 1 monclient: found mon.noname-c 2024-03-29T11:48:14.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:14.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:48:14.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c540ac630 con 0x7f3c54149800 2024-03-29T11:48:14.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: _finish_auth 0 2024-03-29T11:48:14.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3c4c001a10 con 0x7f3c54149800 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: dump: 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:14.021 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:14.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c5a219640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:48:14.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c5a219640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:14.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c4c001c30 con 0x7f3c54149800 2024-03-29T11:48:14.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 1 -- 172.21.15.55:0/3535779347 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3c4c001fc0 con 0x7f3c54149800 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c4bfff640 10 monclient: dump: 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:14.023 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:14.024 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:14.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c5a219640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:14.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.014+0000 7f3c5a219640 4 set_mon_vals no callback set 2024-03-29T11:48:14.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.015+0000 7f3c5a219640 10 monclient: shutdown 2024-03-29T11:48:14.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.015+0000 7f3c5a219640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:14.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.015+0000 7f3c5a219640 1 -- 172.21.15.55:0/3535779347 >> v1:172.21.15.171:6789/0 conn(0x7f3c54149800 legacy=0x7f3c54151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:14.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.016+0000 7f3c5a219640 1 -- 172.21.15.55:0/3535779347 shutdown_connections 2024-03-29T11:48:14.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.016+0000 7f3c5a219640 1 -- 172.21.15.55:0/3535779347 >> 172.21.15.55:0/3535779347 conn(0x7f3c540b3a30 msgr2=0x7f3c540b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:14.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.019+0000 7f3c5a219640 1 -- 172.21.15.55:0/3535779347 shutdown_connections 2024-03-29T11:48:14.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.019+0000 7f3c5a219640 1 -- 172.21.15.55:0/3535779347 wait complete. 2024-03-29T11:48:14.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.020+0000 7f3c5a219640 10 monclient: build_initial_monmap 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.020+0000 7f3c5a219640 10 monclient: monmap: 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:14.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.021+0000 7f3c5a219640 1 librados: starting msgr at 2024-03-29T11:48:14.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.021+0000 7f3c5a219640 1 librados: starting objecter 2024-03-29T11:48:14.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.021+0000 7f3c5a219640 1 Processor -- start 2024-03-29T11:48:14.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.021+0000 7f3c5a219640 1 -- start start 2024-03-29T11:48:14.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.021+0000 7f3c5a219640 1 librados: setting wanted keys 2024-03-29T11:48:14.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.021+0000 7f3c5a219640 1 librados: calling monclient init 2024-03-29T11:48:14.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.021+0000 7f3c5a219640 10 monclient: init 2024-03-29T11:48:14.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.022+0000 7f3c5a219640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:14.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.022+0000 7f3c5a219640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:48:14.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.022+0000 7f3c5a219640 10 monclient(hunting): picked mon.noname-a con 0x7f3c540aa500 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:14.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.022+0000 7f3c5a219640 10 monclient(hunting): picked mon.noname-c con 0x7f3c540aaee0 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:14.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.022+0000 7f3c5a219640 10 monclient(hunting): picked mon.noname-b con 0x7f3c54149800 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:14.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.025+0000 7f3c5a219640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5406e3b0 con 0x7f3c540aa500 2024-03-29T11:48:14.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.025+0000 7f3c5a219640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c54071030 con 0x7f3c54149800 2024-03-29T11:48:14.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.025+0000 7f3c5a219640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c54072760 con 0x7f3c540aaee0 2024-03-29T11:48:14.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.025+0000 7f3c5a219640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:14.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.025+0000 7f3c5a219640 10 monclient(hunting): authenticate will time out at 1307.304927s 2024-03-29T11:48:14.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.025+0000 7f3c59a18640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3c54149800 0x7f3c541e7260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:37079/0 (socket says 172.21.15.55:37079) 2024-03-29T11:48:14.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.025+0000 7f3c59a18640 1 -- 172.21.15.55:0/2505785418 learned_addr learned my addr 172.21.15.55:0/2505785418 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:14.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 client.?.objecter ms_handle_connect 0x7f3c54149800 2024-03-29T11:48:14.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:14.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831964308 0 0) 0x7f3c54071030 con 0x7f3c54149800 2024-03-29T11:48:14.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:14.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:14.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:14.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c38004eb0 con 0x7f3c54149800 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 391053731 0 0) 0x7f3c38004eb0 con 0x7f3c54149800 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c54071030 con 0x7f3c54149800 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3c4c001950 con 0x7f3c54149800 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 monclient(hunting): dump: 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:14.034 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:14.035 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:14.035 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:14.035 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:14.035 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:14.035 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:14.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.026+0000 7f3c49ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:14.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.027+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2272838469 0 0) 0x7f3c54071030 con 0x7f3c54149800 2024-03-29T11:48:14.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.027+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 >> v1:172.21.15.55:6790/0 conn(0x7f3c540aaee0 legacy=0x7f3c54143050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:48:14.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.028+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 >> v1:172.21.15.55:6789/0 conn(0x7f3c540aa500 legacy=0x7f3c540714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:14.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.028+0000 7f3c49ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:48:14.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.028+0000 7f3c49ffb640 1 monclient: found mon.b 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.028+0000 7f3c49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.028+0000 7f3c49ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.028+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54071bd0 con 0x7f3c54149800 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.028+0000 7f3c49ffb640 10 monclient: _finish_auth 0 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.028+0000 7f3c49ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.028+0000 7f3c49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c5a219640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c5a219640 10 monclient: _renew_subs 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c5a219640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c54071ea0 con 0x7f3c54149800 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c5a219640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c5a219640 10 monclient: _renew_subs 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c5a219640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c54143940 con 0x7f3c54149800 2024-03-29T11:48:14.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c5a219640 1 librados: init done 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c49ffb640 10 client.4441.objecter ms_handle_connect 0x7f3c540aa500 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.029+0000 7f3c49ffb640 10 client.4441.objecter resend_mon_ops 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c4c004e80 con 0x7f3c54149800 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c49ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c5a219640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c5a219640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c5a219640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5404f730 con 0x7f3c54149800 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3c4c0052d0 con 0x7f3c54149800 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c49ffb640 10 monclient: dump: 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:14.041 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:14.042 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:14.042 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:14.042 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:14.042 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:14.042 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:14.042 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:14.042 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:14.042 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:14.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.033+0000 7f3c4b7fe640 4 set_mon_vals no callback set 2024-03-29T11:48:14.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.037+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f3c4c0054b0 con 0x7f3c54149800 2024-03-29T11:48:14.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.037+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f3c4c084f60 con 0x7f3c54149800 2024-03-29T11:48:14.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.037+0000 7f3c49ffb640 10 client.4441.objecter ms_dispatch 0x7f3c54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:14.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.037+0000 7f3c49ffb640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:14.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.037+0000 7f3c49ffb640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:14.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.037+0000 7f3c49ffb640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T11:48:14.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.038+0000 7f3c59217640 10 monclient: get_auth_request con 0x7f3c3806dbc0 auth_method 0 2024-03-29T11:48:14.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.040+0000 7f3c49ffb640 10 client.4441.objecter ms_handle_connect 0x7f3c3806dbc0 2024-03-29T11:48:14.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.042+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c4c050840 con 0x7f3c54149800 2024-03-29T11:48:14.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.042+0000 7f3c49ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:14.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.042+0000 7f3c49ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:14.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.376+0000 7f3c5a219640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:14.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.376+0000 7f3c5a219640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:14.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.376+0000 7f3c5a219640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:14.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.376+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c54074d80 con 0x7f3c54149800 2024-03-29T11:48:14.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.381+0000 7f3c49ffb640 1 -- 172.21.15.55:0/2505785418 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f3c4c052770 con 0x7f3c54149800 2024-03-29T11:48:14.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.381+0000 7f3c49ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:14.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.381+0000 7f3c49ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:14.387 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:48:14.387 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 10 librados: watch_flush enter 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 10 librados: watch_flush exit 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 10 client.4441.objecter successfully canceled tick 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 >> v1:172.21.15.171:6816/1463197163 conn(0x7f3c3806dbc0 legacy=0x7f3c38070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 10 monclient: shutdown 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 >> v1:172.21.15.171:6789/0 conn(0x7f3c54149800 legacy=0x7f3c541e7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 shutdown_connections 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 >> 172.21.15.55:0/2505785418 conn(0x7f3c540b3a30 msgr2=0x7f3c54069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 shutdown_connections 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 1 -- 172.21.15.55:0/2505785418 wait complete. 2024-03-29T11:48:14.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:14.385+0000 7f3c5a219640 1 librados: shutdown 2024-03-29T11:48:14.411 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:21.217 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:21.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.368+0000 7faf9b645640 10 monclient: get_monmap_and_config 2024-03-29T11:48:21.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.368+0000 7faf9b645640 10 monclient: build_initial_monmap 2024-03-29T11:48:21.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.368+0000 7faf9b645640 10 monclient: monmap: 2024-03-29T11:48:21.374 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:21.374 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:21.374 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:48:21.369627+0000 2024-03-29T11:48:21.375 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:48:21.369627+0000 2024-03-29T11:48:21.375 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:21.375 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:21.375 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:21.375 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:48:21.375 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:48:21.375 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:21.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.369+0000 7faf9b645640 1 Processor -- start 2024-03-29T11:48:21.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.369+0000 7faf9b645640 1 -- start start 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.369+0000 7faf9b645640 10 monclient: init 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 10 monclient(hunting): picked mon.noname-b con 0x7faf940b0770 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 10 monclient(hunting): picked mon.noname-c con 0x7faf9414a700 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 10 monclient(hunting): picked mon.noname-a con 0x7faf940ac5a0 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940aba70 con 0x7faf940ac5a0 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940aecb0 con 0x7faf9414a700 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940af150 con 0x7faf940b0770 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:21.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.370+0000 7faf9b645640 10 monclient(hunting): authenticate will time out at 1314.650225s 2024-03-29T11:48:21.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.376+0000 7faf993e1640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7faf940b0770 0x7faf94149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:51423/0 (socket says 172.21.15.55:51423) 2024-03-29T11:48:21.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.376+0000 7faf993e1640 1 -- 172.21.15.55:0/2315458056 learned_addr learned my addr 172.21.15.55:0/2315458056 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:21.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 1 -- 172.21.15.55:0/2315458056 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126606213 0 0) 0x7faf940aba70 con 0x7faf940ac5a0 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 1 -- 172.21.15.55:0/2315458056 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf80003520 con 0x7faf940ac5a0 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 1 -- 172.21.15.55:0/2315458056 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2297708145 0 0) 0x7faf80003520 con 0x7faf940ac5a0 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 1 -- 172.21.15.55:0/2315458056 >> v1:172.21.15.55:6790/0 conn(0x7faf940b0770 legacy=0x7faf94149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 1 -- 172.21.15.55:0/2315458056 >> v1:172.21.15.171:6789/0 conn(0x7faf9414a700 legacy=0x7faf940abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 1 monclient: found mon.noname-a 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 1 -- 172.21.15.55:0/2315458056 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf940ac9d0 con 0x7faf940ac5a0 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 10 monclient: _finish_auth 0 2024-03-29T11:48:21.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf9b645640 5 monclient: authenticate success, global_id 4449 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf9b645640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 1 -- 172.21.15.55:0/2315458056 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7faf8c001a10 con 0x7faf940ac5a0 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.377+0000 7faf8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.378+0000 7faf8bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.378+0000 7faf8bfff640 10 monclient: dump: 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:21.390 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.378+0000 7faf9b645640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.378+0000 7faf8bfff640 1 -- 172.21.15.55:0/2315458056 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf8c001ca0 con 0x7faf940ac5a0 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.378+0000 7faf8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.378+0000 7faf9b645640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.378+0000 7faf9b645640 4 set_mon_vals no callback set 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.379+0000 7faf9b645640 10 monclient: shutdown 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.379+0000 7faf9b645640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.379+0000 7faf9b645640 1 -- 172.21.15.55:0/2315458056 >> v1:172.21.15.55:6789/0 conn(0x7faf940ac5a0 legacy=0x7faf94151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.382+0000 7faf9b645640 1 -- 172.21.15.55:0/2315458056 shutdown_connections 2024-03-29T11:48:21.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.382+0000 7faf9b645640 1 -- 172.21.15.55:0/2315458056 >> 172.21.15.55:0/2315458056 conn(0x7faf9406b660 msgr2=0x7faf9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:21.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.383+0000 7faf9b645640 1 -- 172.21.15.55:0/2315458056 shutdown_connections 2024-03-29T11:48:21.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.386+0000 7faf9b645640 1 -- 172.21.15.55:0/2315458056 wait complete. 2024-03-29T11:48:21.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.387+0000 7faf9b645640 10 monclient: build_initial_monmap 2024-03-29T11:48:21.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.387+0000 7faf9b645640 10 monclient: monmap: 2024-03-29T11:48:21.393 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:21.393 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:21.393 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:48:21.393 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:48:21.394 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:21.394 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:21.394 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:21.394 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:48:21.394 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:48:21.394 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:21.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.388+0000 7faf9b645640 1 librados: starting msgr at 2024-03-29T11:48:21.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.388+0000 7faf9b645640 1 librados: starting objecter 2024-03-29T11:48:21.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.388+0000 7faf9b645640 1 Processor -- start 2024-03-29T11:48:21.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.389+0000 7faf9b645640 1 -- start start 2024-03-29T11:48:21.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.389+0000 7faf9b645640 1 librados: setting wanted keys 2024-03-29T11:48:21.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.389+0000 7faf9b645640 1 librados: calling monclient init 2024-03-29T11:48:21.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.389+0000 7faf9b645640 10 monclient: init 2024-03-29T11:48:21.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:21.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:48:21.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 10 monclient(hunting): picked mon.noname-b con 0x7faf940ac5a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:21.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 10 monclient(hunting): picked mon.noname-c con 0x7faf940b0770 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:21.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 10 monclient(hunting): picked mon.noname-a con 0x7faf9414a700 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:21.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf941286c0 con 0x7faf9414a700 2024-03-29T11:48:21.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf94127cb0 con 0x7faf940ac5a0 2024-03-29T11:48:21.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf94147b30 con 0x7faf940b0770 2024-03-29T11:48:21.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:21.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.390+0000 7faf9b645640 10 monclient(hunting): authenticate will time out at 1314.670516s 2024-03-29T11:48:21.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.393+0000 7faf993e1640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7faf940ac5a0 0x7faf94128bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:41519/0 (socket says 172.21.15.55:41519) 2024-03-29T11:48:21.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf993e1640 1 -- 172.21.15.55:0/3982488623 learned_addr learned my addr 172.21.15.55:0/3982488623 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:21.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 10 client.?.objecter ms_handle_connect 0x7faf940ac5a0 2024-03-29T11:48:21.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:21.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612136598 0 0) 0x7faf94127cb0 con 0x7faf940ac5a0 2024-03-29T11:48:21.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:21.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:21.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:21.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:48:21.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf78004eb0 con 0x7faf940ac5a0 2024-03-29T11:48:21.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 10 client.?.objecter ms_handle_connect 0x7faf9414a700 2024-03-29T11:48:21.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.394+0000 7faf89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:21.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.396+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1522367006 0 0) 0x7faf78004eb0 con 0x7faf940ac5a0 2024-03-29T11:48:21.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.396+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf94127cb0 con 0x7faf940ac5a0 2024-03-29T11:48:21.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.396+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7faf90001930 con 0x7faf940ac5a0 2024-03-29T11:48:21.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.396+0000 7faf89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:21.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.396+0000 7faf89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:21.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.396+0000 7faf89ffb640 10 monclient(hunting): dump: 2024-03-29T11:48:21.403 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.396+0000 7faf89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3558765440 0 0) 0x7faf94127cb0 con 0x7faf940ac5a0 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 >> v1:172.21.15.55:6790/0 conn(0x7faf940b0770 legacy=0x7faf94146e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 >> v1:172.21.15.55:6789/0 conn(0x7faf9414a700 legacy=0x7faf941db670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:48:21.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 1 monclient: found mon.b 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf94147cc0 con 0x7faf940ac5a0 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 10 monclient: _finish_auth 0 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf9b645640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf9b645640 10 monclient: _renew_subs 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf9b645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.397+0000 7faf9b645640 1 -- 172.21.15.55:0/3982488623 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf94147530 con 0x7faf940ac5a0 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf900047e0 con 0x7faf940ac5a0 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7faf90004bb0 con 0x7faf940ac5a0 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf89ffb640 10 monclient: dump: 2024-03-29T11:48:21.405 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:21.406 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:21.406 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:21.406 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:21.406 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:21.406 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:21.406 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:21.406 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:21.406 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:21.406 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:21.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf9b645640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:21.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf9b645640 10 monclient: _renew_subs 2024-03-29T11:48:21.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf9b645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:21.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf9b645640 1 -- 172.21.15.55:0/3982488623 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf941dbd80 con 0x7faf940ac5a0 2024-03-29T11:48:21.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf9b645640 1 librados: init done 2024-03-29T11:48:21.409 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.403+0000 7faf7f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:21.409 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.398+0000 7faf8b7fe640 4 set_mon_vals no callback set 2024-03-29T11:48:21.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.405+0000 7faf7f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:21.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.405+0000 7faf7f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:21.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.405+0000 7faf7f7fe640 1 -- 172.21.15.55:0/3982488623 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf940ab560 con 0x7faf940ac5a0 2024-03-29T11:48:21.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.405+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7faf90004d10 con 0x7faf940ac5a0 2024-03-29T11:48:21.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.406+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7faf90084bd0 con 0x7faf940ac5a0 2024-03-29T11:48:21.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.407+0000 7faf89ffb640 10 client.4444.objecter ms_dispatch 0x7faf94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:21.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.407+0000 7faf89ffb640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:21.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.407+0000 7faf89ffb640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:21.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.407+0000 7faf89ffb640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-29T11:48:21.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.407+0000 7faf98be0640 10 monclient: get_auth_request con 0x7faf7806dd50 auth_method 0 2024-03-29T11:48:21.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.409+0000 7faf89ffb640 10 client.4444.objecter ms_handle_connect 0x7faf7806dd50 2024-03-29T11:48:21.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.414+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf900504b0 con 0x7faf940ac5a0 2024-03-29T11:48:21.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.414+0000 7faf89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:21.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.414+0000 7faf89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:21.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.746+0000 7faf7f7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:21.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.746+0000 7faf7f7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:21.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.746+0000 7faf7f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:21.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.746+0000 7faf7f7fe640 1 -- 172.21.15.55:0/3982488623 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf94128520 con 0x7faf940ac5a0 2024-03-29T11:48:21.757 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:48:21.757 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:21.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.748+0000 7faf89ffb640 1 -- 172.21.15.55:0/3982488623 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7faf94128520 con 0x7faf940ac5a0 2024-03-29T11:48:21.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.748+0000 7faf89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:21.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.748+0000 7faf89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:21.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.754+0000 7faf7f7fe640 10 librados: watch_flush enter 2024-03-29T11:48:21.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.755+0000 7faf7f7fe640 10 librados: watch_flush exit 2024-03-29T11:48:21.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.755+0000 7faf7f7fe640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-29T11:48:21.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.755+0000 7faf7f7fe640 10 client.4444.objecter successfully canceled tick 2024-03-29T11:48:21.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.755+0000 7faf7f7fe640 1 -- 172.21.15.55:0/3982488623 >> v1:172.21.15.171:6816/1463197163 conn(0x7faf7806dd50 legacy=0x7faf78070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.755+0000 7faf7f7fe640 10 monclient: shutdown 2024-03-29T11:48:21.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.755+0000 7faf7f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:21.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.755+0000 7faf7f7fe640 1 -- 172.21.15.55:0/3982488623 >> v1:172.21.15.171:6789/0 conn(0x7faf940ac5a0 legacy=0x7faf94128bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.755+0000 7faf7f7fe640 1 -- 172.21.15.55:0/3982488623 shutdown_connections 2024-03-29T11:48:21.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.755+0000 7faf7f7fe640 1 -- 172.21.15.55:0/3982488623 >> 172.21.15.55:0/3982488623 conn(0x7faf9406b660 msgr2=0x7faf94149c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:21.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.756+0000 7faf7f7fe640 1 -- 172.21.15.55:0/3982488623 shutdown_connections 2024-03-29T11:48:21.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.756+0000 7faf7f7fe640 1 -- 172.21.15.55:0/3982488623 wait complete. 2024-03-29T11:48:21.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:21.758+0000 7faf7f7fe640 1 librados: shutdown 2024-03-29T11:48:21.781 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:28.588 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:28.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.751+0000 7f0fffe96640 10 monclient: get_monmap_and_config 2024-03-29T11:48:28.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.751+0000 7f0fffe96640 10 monclient: build_initial_monmap 2024-03-29T11:48:28.756 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.751+0000 7f0fffe96640 10 monclient: monmap: 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:48:28.751736+0000 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:48:28.751736+0000 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:48:28.757 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:28.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.752+0000 7f0fffe96640 1 Processor -- start 2024-03-29T11:48:28.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.752+0000 7f0fffe96640 1 -- start start 2024-03-29T11:48:28.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.752+0000 7f0fffe96640 10 monclient: init 2024-03-29T11:48:28.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.753+0000 7f0fffe96640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:28.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0fffe96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0fffe96640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff8071de0 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0fffe96640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff80721c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0fffe96640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff8145ae0 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0fffe96640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff8145650 con 0x7f0ff8145ae0 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0fffe96640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff80ae260 con 0x7f0ff80721c0 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0fffe96640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff81420d0 con 0x7f0ff8071de0 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0fffe96640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0fffe96640 10 monclient(hunting): authenticate will time out at 1322.032770s 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0ffd431640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0ff80721c0 0x7f0ff813f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:55346/0 (socket says 172.21.15.55:55346) 2024-03-29T11:48:28.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.754+0000 7f0ffd431640 1 -- 172.21.15.55:0/3213653965 learned_addr learned my addr 172.21.15.55:0/3213653965 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231585040 0 0) 0x7f0ff80ae260 con 0x7f0ff80721c0 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): _negotiate 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe4003520 con 0x7f0ff80721c0 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157687465 0 0) 0x7f0ff81420d0 con 0x7f0ff8071de0 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): _negotiate 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:28.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff80ae260 con 0x7f0ff8071de0 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297143551 0 0) 0x7f0ff8145650 con 0x7f0ff8145ae0 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): _negotiate 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.755+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff81420d0 con 0x7f0ff8145ae0 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822093539 0 0) 0x7f0fe4003520 con 0x7f0ff80721c0 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 >> v1:172.21.15.55:6790/0 conn(0x7f0ff8071de0 legacy=0x7f0ff80aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 >> v1:172.21.15.55:6789/0 conn(0x7f0ff8145ae0 legacy=0x7f0ff8145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 10 monclient: _finish_hunting 0 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 1 monclient: found mon.noname-c 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:28.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff80b0590 con 0x7f0ff80721c0 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 10 monclient: _finish_auth 0 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 10 monclient: _check_auth_tickets 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0fec001690 con 0x7f0ff80721c0 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0ffcc30640 10 monclient: dump: 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:28.763 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:28.764 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:28.764 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:28.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0fffe96640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:48:28.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.756+0000 7f0fffe96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.757+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fec001870 con 0x7f0ff80721c0 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.757+0000 7f0ffcc30640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.757+0000 7f0fffe96640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.757+0000 7f0fffe96640 4 set_mon_vals no callback set 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.757+0000 7f0ffcc30640 1 -- 172.21.15.55:0/3213653965 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0fec001c00 con 0x7f0ff80721c0 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.757+0000 7f0ffcc30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.757+0000 7f0ffcc30640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.757+0000 7f0ffcc30640 10 monclient: dump: 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:28.765 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.758+0000 7f0fffe96640 10 monclient: shutdown 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.758+0000 7f0fffe96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.758+0000 7f0fffe96640 1 -- 172.21.15.55:0/3213653965 >> v1:172.21.15.171:6789/0 conn(0x7f0ff80721c0 legacy=0x7f0ff813f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.758+0000 7f0fffe96640 1 -- 172.21.15.55:0/3213653965 shutdown_connections 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.758+0000 7f0fffe96640 1 -- 172.21.15.55:0/3213653965 >> 172.21.15.55:0/3213653965 conn(0x7f0ff806b660 msgr2=0x7f0ff806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.758+0000 7f0fffe96640 1 -- 172.21.15.55:0/3213653965 shutdown_connections 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.758+0000 7f0fffe96640 1 -- 172.21.15.55:0/3213653965 wait complete. 2024-03-29T11:48:28.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.759+0000 7f0fffe96640 10 monclient: build_initial_monmap 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.759+0000 7f0fffe96640 10 monclient: monmap: 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.759+0000 7f0fffe96640 1 librados: starting msgr at 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.759+0000 7f0fffe96640 1 librados: starting objecter 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.760+0000 7f0fffe96640 1 Processor -- start 2024-03-29T11:48:28.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.760+0000 7f0fffe96640 1 -- start start 2024-03-29T11:48:28.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.760+0000 7f0fffe96640 1 librados: setting wanted keys 2024-03-29T11:48:28.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.760+0000 7f0fffe96640 1 librados: calling monclient init 2024-03-29T11:48:28.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.760+0000 7f0fffe96640 10 monclient: init 2024-03-29T11:48:28.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.760+0000 7f0fffe96640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:28.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.760+0000 7f0fffe96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:48:28.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fffe96640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff8071de0 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:28.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fffe96640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff8145ae0 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0ffdc32640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0ff8071de0 0x7f0ff80ac420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:57406/0 (socket says 172.21.15.55:57406) 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0ffe433640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f0ff813ad60 0x7f0ff81e1f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:46722/0 (socket says 172.21.15.55:46722) 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0ffd431640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0ff8145ae0 0x7f0ff813c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:59172/0 (socket says 172.21.15.55:59172) 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fffe96640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff813ad60 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0ffdc32640 1 -- 172.21.15.55:0/816806296 learned_addr learned my addr 172.21.15.55:0/816806296 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff813a490 con 0x7f0ff8145ae0 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fea7fc640 10 client.?.objecter ms_handle_connect 0x7f0ff813ad60 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fea7fc640 10 client.?.objecter ms_handle_connect 0x7f0ff8071de0 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff813b7c0 con 0x7f0ff8071de0 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff813ba30 con 0x7f0ff813ad60 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fffe96640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:28.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fffe96640 10 monclient(hunting): authenticate will time out at 1322.040459s 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fea7fc640 10 client.?.objecter ms_handle_connect 0x7f0ff8145ae0 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0ffe433640 1 -- 172.21.15.55:0/816806296 >> v1:172.21.15.55:6790/0 conn(0x7f0ff813ad60 legacy=0x7f0ff81e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0ffe433640 1 -- 172.21.15.55:0/816806296 >> v1:172.21.15.55:6790/0 conn(0x7f0ff813ad60 legacy=0x7f0ff81e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0ffe433640 1 --1- 172.21.15.55:0/816806296 >> v1:172.21.15.55:6790/0 conn(0x7f0ff813ad60 0x7f0ff81e1f70 :-1 s=OPENED pgs=588735 cs=1 l=1).handle_message read tag failed 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0ffe433640 1 --1- 172.21.15.55:0/816806296 >> v1:172.21.15.55:6790/0 conn(0x7f0ff813ad60 0x7f0ff81e1f70 :-1 s=OPENED pgs=588735 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.761+0000 7f0fea7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.55:6790/0 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773289290 0 0) 0x7f0ff813b7c0 con 0x7f0ff8071de0 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fdc004eb0 con 0x7f0ff8071de0 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256220863 0 0) 0x7f0ff813a490 con 0x7f0ff8145ae0 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:28.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff813b7c0 con 0x7f0ff8145ae0 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2138497371 0 0) 0x7f0fdc004eb0 con 0x7f0ff8071de0 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ff813a490 con 0x7f0ff8071de0 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0ff40018c0 con 0x7f0ff8071de0 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): dump: 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:28.773 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.762+0000 7f0fea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1145753936 0 0) 0x7f0ff813a490 con 0x7f0ff8071de0 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 >> v1:172.21.15.55:6790/0 conn(0x7f0ff813ad60 legacy=0x7f0ff81e1f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 >> v1:172.21.15.55:6789/0 conn(0x7f0ff8145ae0 legacy=0x7f0ff813c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 1 monclient: found mon.b 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff813bd10 con 0x7f0ff8071de0 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:48:28.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fffe96640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fffe96640 10 monclient: _renew_subs 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fffe96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff813bff0 con 0x7f0ff8071de0 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fffe96640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fffe96640 10 monclient: _renew_subs 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fffe96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff8139fe0 con 0x7f0ff8071de0 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.763+0000 7f0fffe96640 1 librados: init done 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.764+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ff4001d50 con 0x7f0ff8071de0 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.764+0000 7f0fea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.764+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0ff40037b0 con 0x7f0ff8071de0 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.764+0000 7f0fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.764+0000 7f0fea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:28.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.764+0000 7f0fea7fc640 10 monclient: dump: 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.764+0000 7f0feb7fe640 4 set_mon_vals no callback set 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.765+0000 7f0fffe96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.765+0000 7f0fffe96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.765+0000 7f0fffe96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:28.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.765+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe0005080 con 0x7f0ff8071de0 2024-03-29T11:48:28.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.766+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f0ff4003910 con 0x7f0ff8071de0 2024-03-29T11:48:28.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.767+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f0ff4084920 con 0x7f0ff8071de0 2024-03-29T11:48:28.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.767+0000 7f0fea7fc640 10 client.4450.objecter ms_dispatch 0x7f0ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:28.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.767+0000 7f0fea7fc640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:28.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.767+0000 7f0fea7fc640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:28.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.767+0000 7f0fea7fc640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-29T11:48:28.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.767+0000 7f0ffd431640 10 monclient: get_auth_request con 0x7f0fdc06dd40 auth_method 0 2024-03-29T11:48:28.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.767+0000 7f0fea7fc640 10 client.4450.objecter ms_handle_connect 0x7f0fdc06dd40 2024-03-29T11:48:28.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.773+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ff4050030 con 0x7f0ff8071de0 2024-03-29T11:48:28.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.773+0000 7f0fea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:28.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:28.773+0000 7f0fea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:29.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.156+0000 7f0fffe96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:29.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.156+0000 7f0fffe96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:29.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.157+0000 7f0fffe96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:48:29.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.157+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fe0005330 con 0x7f0ff8071de0 2024-03-29T11:48:29.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.158+0000 7f0fea7fc640 1 -- 172.21.15.55:0/816806296 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f0ff4052130 con 0x7f0ff8071de0 2024-03-29T11:48:29.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.158+0000 7f0fea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:29.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.158+0000 7f0fea7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:29.165 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:48:29.165 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:29.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 10 librados: watch_flush enter 2024-03-29T11:48:29.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 10 librados: watch_flush exit 2024-03-29T11:48:29.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-29T11:48:29.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 10 client.4450.objecter successfully canceled tick 2024-03-29T11:48:29.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 >> v1:172.21.15.171:6816/1463197163 conn(0x7f0fdc06dd40 legacy=0x7f0fdc070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:29.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 10 monclient: shutdown 2024-03-29T11:48:29.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:29.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 >> v1:172.21.15.171:6789/0 conn(0x7f0ff8071de0 legacy=0x7f0ff80ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:29.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 shutdown_connections 2024-03-29T11:48:29.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 >> 172.21.15.55:0/816806296 conn(0x7f0ff806b660 msgr2=0x7f0ff80a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:29.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 shutdown_connections 2024-03-29T11:48:29.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 1 -- 172.21.15.55:0/816806296 wait complete. 2024-03-29T11:48:29.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:29.162+0000 7f0fffe96640 1 librados: shutdown 2024-03-29T11:48:29.185 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:35.992 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:36.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.138+0000 7f0e7cf38640 10 monclient: get_monmap_and_config 2024-03-29T11:48:36.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.138+0000 7f0e7cf38640 10 monclient: build_initial_monmap 2024-03-29T11:48:36.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.139+0000 7f0e7cf38640 10 monclient: monmap: 2024-03-29T11:48:36.147 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:36.147 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:36.147 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:48:36.140173+0000 2024-03-29T11:48:36.147 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:48:36.140173+0000 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.139+0000 7f0e7cf38640 1 Processor -- start 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.139+0000 7f0e7cf38640 1 -- start start 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.139+0000 7f0e7cf38640 10 monclient: init 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 10 monclient(hunting): picked mon.noname-b con 0x7f0e781457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 10 monclient(hunting): picked mon.noname-a con 0x7f0e7813f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:36.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 10 monclient(hunting): picked mon.noname-c con 0x7f0e780a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:36.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e780aaef0 con 0x7f0e7813f650 2024-03-29T11:48:36.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e78145bd0 con 0x7f0e780a9d70 2024-03-29T11:48:36.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e78145eb0 con 0x7f0e781457f0 2024-03-29T11:48:36.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:36.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7cf38640 10 monclient(hunting): authenticate will time out at 1329.420180s 2024-03-29T11:48:36.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e75d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0e7813f650 0x7f0e780b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:55286/0 (socket says 172.21.15.55:55286) 2024-03-29T11:48:36.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e75d9b640 1 -- 172.21.15.55:0/1975332550 learned_addr learned my addr 172.21.15.55:0/1975332550 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:36.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.140+0000 7f0e7659c640 1 --1- 172.21.15.55:0/1975332550 >> v1:172.21.15.55:6790/0 conn(0x7f0e781457f0 0x7f0e7813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:46508/0 (socket says 172.21.15.55:46508) 2024-03-29T11:48:36.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751182541 0 0) 0x7f0e78145bd0 con 0x7f0e780a9d70 2024-03-29T11:48:36.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 10 monclient(hunting): _negotiate 2024-03-29T11:48:36.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:36.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:36.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e60003520 con 0x7f0e780a9d70 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402103357 0 0) 0x7f0e78145eb0 con 0x7f0e781457f0 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 10 monclient(hunting): _negotiate 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.142+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e60003840 con 0x7f0e781457f0 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925844856 0 0) 0x7f0e780aaef0 con 0x7f0e7813f650 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 10 monclient(hunting): _negotiate 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e78145eb0 con 0x7f0e7813f650 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 949773437 0 0) 0x7f0e60003840 con 0x7f0e781457f0 2024-03-29T11:48:36.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 >> v1:172.21.15.171:6789/0 conn(0x7f0e780a9d70 legacy=0x7f0e7814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 >> v1:172.21.15.55:6789/0 conn(0x7f0e7813f650 legacy=0x7f0e780b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 10 monclient: _finish_hunting 0 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 1 monclient: found mon.noname-b 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e78146340 con 0x7f0e781457f0 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.143+0000 7f0e7559a640 10 monclient: _finish_auth 0 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7559a640 10 monclient: _check_auth_tickets 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0e68001670 con 0x7f0e781457f0 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7559a640 10 monclient: dump: 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:36.153 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7cf38640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7cf38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e68001890 con 0x7f0e781457f0 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7cf38640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.144+0000 7f0e7cf38640 4 set_mon_vals no callback set 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7cf38640 10 monclient: shutdown 2024-03-29T11:48:36.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7559a640 1 -- 172.21.15.55:0/1975332550 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0e68001c20 con 0x7f0e781457f0 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7559a640 10 monclient: dump: 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7cf38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1975332550 >> v1:172.21.15.55:6790/0 conn(0x7f0e781457f0 legacy=0x7f0e7813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:36.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1975332550 shutdown_connections 2024-03-29T11:48:36.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1975332550 >> 172.21.15.55:0/1975332550 conn(0x7f0e78066a00 msgr2=0x7f0e78066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:36.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1975332550 shutdown_connections 2024-03-29T11:48:36.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.145+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1975332550 wait complete. 2024-03-29T11:48:36.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.147+0000 7f0e7cf38640 10 monclient: build_initial_monmap 2024-03-29T11:48:36.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.147+0000 7f0e7cf38640 10 monclient: monmap: 2024-03-29T11:48:36.160 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:36.160 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.147+0000 7f0e7cf38640 1 librados: starting msgr at 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.147+0000 7f0e7cf38640 1 librados: starting objecter 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.147+0000 7f0e7cf38640 1 Processor -- start 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.147+0000 7f0e7cf38640 1 -- start start 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.147+0000 7f0e7cf38640 1 librados: setting wanted keys 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.147+0000 7f0e7cf38640 1 librados: calling monclient init 2024-03-29T11:48:36.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.147+0000 7f0e7cf38640 10 monclient: init 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.148+0000 7f0e7cf38640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.148+0000 7f0e7cf38640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.148+0000 7f0e7cf38640 10 monclient(hunting): picked mon.noname-a con 0x7f0e780a9d70 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e7cf38640 10 monclient(hunting): picked mon.noname-b con 0x7f0e7813f650 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e7659c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0e780a9d70 0x7f0e78137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:55746/0 (socket says 172.21.15.55:55746) 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e7659c640 1 -- 172.21.15.55:0/1183277996 learned_addr learned my addr 172.21.15.55:0/1183277996 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e7cf38640 10 monclient(hunting): picked mon.noname-c con 0x7f0e781457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 client.?.objecter ms_handle_connect 0x7f0e780a9d70 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7813b310 con 0x7f0e780a9d70 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e78137c40 con 0x7f0e7813f650 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e78137dd0 con 0x7f0e781457f0 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e7cf38640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e7cf38640 10 monclient(hunting): authenticate will time out at 1329.429350s 2024-03-29T11:48:36.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 client.?.objecter ms_handle_connect 0x7f0e7813f650 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 client.?.objecter ms_handle_connect 0x7f0e781457f0 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010003028 0 0) 0x7f0e7813b310 con 0x7f0e780a9d70 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.149+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e58004eb0 con 0x7f0e780a9d70 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779993316 0 0) 0x7f0e78137dd0 con 0x7f0e781457f0 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:48:36.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7813b310 con 0x7f0e781457f0 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2703305686 0 0) 0x7f0e58004eb0 con 0x7f0e780a9d70 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e78137dd0 con 0x7f0e780a9d70 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0e68001c60 con 0x7f0e780a9d70 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): dump: 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.164 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934388581 0 0) 0x7f0e78137c40 con 0x7f0e7813f650 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.150+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e58004eb0 con 0x7f0e7813f650 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.151+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2577799176 0 0) 0x7f0e7813b310 con 0x7f0e781457f0 2024-03-29T11:48:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.151+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e78137c40 con 0x7f0e781457f0 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.151+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0e70001c30 con 0x7f0e781457f0 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.151+0000 7f0e66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.151+0000 7f0e66ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.151+0000 7f0e66ffd640 10 monclient(hunting): dump: 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:36.166 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4121760563 0 0) 0x7f0e58004eb0 con 0x7f0e7813f650 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7813b310 con 0x7f0e7813f650 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3990726495 0 0) 0x7f0e78137c40 con 0x7f0e781457f0 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 >> v1:172.21.15.171:6789/0 conn(0x7f0e7813f650 legacy=0x7f0e781dbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 >> v1:172.21.15.55:6789/0 conn(0x7f0e780a9d70 legacy=0x7f0e78137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 1 monclient: found mon.c 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e78137f60 con 0x7f0e781457f0 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:48:36.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e7cf38640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e7cf38640 10 monclient: _renew_subs 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e7cf38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7813a210 con 0x7f0e781457f0 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e7cf38640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e7cf38640 10 monclient: _renew_subs 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e7cf38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e78141cb0 con 0x7f0e781457f0 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e7cf38640 1 librados: init done 2024-03-29T11:48:36.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e70002040 con 0x7f0e781457f0 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.152+0000 7f0e66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.153+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0e70003b10 con 0x7f0e781457f0 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.153+0000 7f0e74d99640 4 set_mon_vals no callback set 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.153+0000 7f0e66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.153+0000 7f0e66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.153+0000 7f0e66ffd640 10 monclient: dump: 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:36.169 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:36.170 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:36.170 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:36.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.154+0000 7f0e7cf38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:36.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.154+0000 7f0e7cf38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:36.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.154+0000 7f0e7cf38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:48:36.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.154+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5c005080 con 0x7f0e781457f0 2024-03-29T11:48:36.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.155+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f0e70019ae0 con 0x7f0e781457f0 2024-03-29T11:48:36.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.156+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f0e70083910 con 0x7f0e781457f0 2024-03-29T11:48:36.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.156+0000 7f0e66ffd640 10 client.4466.objecter ms_dispatch 0x7f0e78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:36.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.156+0000 7f0e66ffd640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:36.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.156+0000 7f0e66ffd640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:36.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.156+0000 7f0e66ffd640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T11:48:36.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.156+0000 7f0e7659c640 10 monclient: get_auth_request con 0x7f0e5806e1c0 auth_method 0 2024-03-29T11:48:36.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.157+0000 7f0e66ffd640 10 client.4466.objecter ms_handle_connect 0x7f0e5806e1c0 2024-03-29T11:48:36.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.165+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e7004f1f0 con 0x7f0e781457f0 2024-03-29T11:48:36.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.165+0000 7f0e66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:36.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.165+0000 7f0e66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:36.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.577+0000 7f0e7cf38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:36.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.577+0000 7f0e7cf38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:36.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.577+0000 7f0e7cf38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:48:36.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.577+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e5c005330 con 0x7f0e781457f0 2024-03-29T11:48:36.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.578+0000 7f0e66ffd640 1 -- 172.21.15.55:0/1183277996 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f0e70051120 con 0x7f0e781457f0 2024-03-29T11:48:36.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.578+0000 7f0e66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:36.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.578+0000 7f0e66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:36.585 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:48:36.585 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:36.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.581+0000 7f0e7cf38640 10 librados: watch_flush enter 2024-03-29T11:48:36.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 10 librados: watch_flush exit 2024-03-29T11:48:36.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T11:48:36.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 10 client.4466.objecter successfully canceled tick 2024-03-29T11:48:36.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 >> v1:172.21.15.171:6816/1463197163 conn(0x7f0e5806e1c0 legacy=0x7f0e58070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:36.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 10 monclient: shutdown 2024-03-29T11:48:36.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:36.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 >> v1:172.21.15.55:6790/0 conn(0x7f0e781457f0 legacy=0x7f0e781e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:36.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 shutdown_connections 2024-03-29T11:48:36.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 >> 172.21.15.55:0/1183277996 conn(0x7f0e78066a00 msgr2=0x7f0e780b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:36.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 shutdown_connections 2024-03-29T11:48:36.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 1 -- 172.21.15.55:0/1183277996 wait complete. 2024-03-29T11:48:36.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:36.582+0000 7f0e7cf38640 1 librados: shutdown 2024-03-29T11:48:36.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:43.410 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:43.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.556+0000 7ffbcd975640 10 monclient: get_monmap_and_config 2024-03-29T11:48:43.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.556+0000 7ffbcd975640 10 monclient: build_initial_monmap 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.556+0000 7ffbcd975640 10 monclient: monmap: 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:48:43.557393+0000 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:48:43.557393+0000 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:43.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.558+0000 7ffbcd975640 1 Processor -- start 2024-03-29T11:48:43.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.558+0000 7ffbcd975640 1 -- start start 2024-03-29T11:48:43.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.558+0000 7ffbcd975640 10 monclient: init 2024-03-29T11:48:43.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.559+0000 7ffbcd975640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:43.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.559+0000 7ffbcd975640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:48:43.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.559+0000 7ffbcd975640 10 monclient(hunting): picked mon.noname-a con 0x7ffbc80b3f30 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:43.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.562+0000 7ffbcd975640 10 monclient(hunting): picked mon.noname-b con 0x7ffbc80ae220 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:43.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.562+0000 7ffbcd975640 10 monclient(hunting): picked mon.noname-c con 0x7ffbc813b2c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:43.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.562+0000 7ffbcd975640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc8142490 con 0x7ffbc80b3f30 2024-03-29T11:48:43.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.562+0000 7ffbcd975640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc80ac410 con 0x7ffbc813b2c0 2024-03-29T11:48:43.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.562+0000 7ffbcd975640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc80afa40 con 0x7ffbc80ae220 2024-03-29T11:48:43.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.562+0000 7ffbcd975640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:43.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.562+0000 7ffbcd975640 10 monclient(hunting): authenticate will time out at 1336.842070s 2024-03-29T11:48:43.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.566+0000 7ffbcd174640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ffbc813b2c0 0x7ffbc813b710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:36797/0 (socket says 172.21.15.55:36797) 2024-03-29T11:48:43.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.566+0000 7ffbcd174640 1 -- 172.21.15.55:0/2573662415 learned_addr learned my addr 172.21.15.55:0/2573662415 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:43.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.566+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755498680 0 0) 0x7ffbc80ac410 con 0x7ffbc813b2c0 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.566+0000 7ffbc77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb0003520 con 0x7ffbc813b2c0 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195105683 0 0) 0x7ffbc8142490 con 0x7ffbc80b3f30 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:48:43.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc80ac410 con 0x7ffbc80b3f30 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3431298913 0 0) 0x7ffbb0003520 con 0x7ffbc813b2c0 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 >> v1:172.21.15.55:6790/0 conn(0x7ffbc80ae220 legacy=0x7ffbc8141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 >> v1:172.21.15.55:6789/0 conn(0x7ffbc80b3f30 legacy=0x7ffbc80a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 1 monclient: found mon.noname-c 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc80aff90 con 0x7ffbc813b2c0 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient: _finish_auth 0 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.567+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ffbb8001690 con 0x7ffbc813b2c0 2024-03-29T11:48:43.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.568+0000 7ffbc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.568+0000 7ffbc77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.568+0000 7ffbc77fe640 10 monclient: dump: 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:43.576 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:43.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.569+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbb80018b0 con 0x7ffbc813b2c0 2024-03-29T11:48:43.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.570+0000 7ffbc77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:43.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.570+0000 7ffbc77fe640 1 -- 172.21.15.55:0/2573662415 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ffbb8001c40 con 0x7ffbc813b2c0 2024-03-29T11:48:43.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.570+0000 7ffbc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:43.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.570+0000 7ffbc77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.570+0000 7ffbc77fe640 10 monclient: dump: 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.570+0000 7ffbcd975640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.570+0000 7ffbcd975640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:43.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.570+0000 7ffbcd975640 4 set_mon_vals no callback set 2024-03-29T11:48:43.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.573+0000 7ffbcd975640 10 monclient: shutdown 2024-03-29T11:48:43.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.573+0000 7ffbcd975640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:43.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.573+0000 7ffbcd975640 1 -- 172.21.15.55:0/2573662415 >> v1:172.21.15.171:6789/0 conn(0x7ffbc813b2c0 legacy=0x7ffbc813b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:43.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.573+0000 7ffbcd975640 1 -- 172.21.15.55:0/2573662415 shutdown_connections 2024-03-29T11:48:43.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.573+0000 7ffbcd975640 1 -- 172.21.15.55:0/2573662415 >> 172.21.15.55:0/2573662415 conn(0x7ffbc80673e0 msgr2=0x7ffbc80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:43.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.575+0000 7ffbcd975640 1 -- 172.21.15.55:0/2573662415 shutdown_connections 2024-03-29T11:48:43.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.575+0000 7ffbcd975640 1 -- 172.21.15.55:0/2573662415 wait complete. 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.576+0000 7ffbcd975640 10 monclient: build_initial_monmap 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.576+0000 7ffbcd975640 10 monclient: monmap: 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.576+0000 7ffbcd975640 1 librados: starting msgr at 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.576+0000 7ffbcd975640 1 librados: starting objecter 2024-03-29T11:48:43.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.577+0000 7ffbcd975640 1 Processor -- start 2024-03-29T11:48:43.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.577+0000 7ffbcd975640 1 -- start start 2024-03-29T11:48:43.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.577+0000 7ffbcd975640 1 librados: setting wanted keys 2024-03-29T11:48:43.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.577+0000 7ffbcd975640 1 librados: calling monclient init 2024-03-29T11:48:43.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.577+0000 7ffbcd975640 10 monclient: init 2024-03-29T11:48:43.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.578+0000 7ffbcd975640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:43.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.578+0000 7ffbcd975640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:48:43.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.578+0000 7ffbcd975640 10 monclient(hunting): picked mon.noname-a con 0x7ffbc80ae220 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:43.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.581+0000 7ffbcd975640 10 monclient(hunting): picked mon.noname-c con 0x7ffbc80b3f30 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:43.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.583+0000 7ffbcd975640 10 monclient(hunting): picked mon.noname-b con 0x7ffbc813b2c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:43.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.583+0000 7ffbcd975640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc8133d30 con 0x7ffbc80ae220 2024-03-29T11:48:43.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.583+0000 7ffbcd975640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc81db5b0 con 0x7ffbc813b2c0 2024-03-29T11:48:43.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.583+0000 7ffbcd975640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc813be30 con 0x7ffbc80b3f30 2024-03-29T11:48:43.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.583+0000 7ffbcd975640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:43.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.583+0000 7ffbcd975640 10 monclient(hunting): authenticate will time out at 1336.863016s 2024-03-29T11:48:43.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.583+0000 7ffbcc973640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ffbc80ae220 0x7ffbc8130b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42761/0 (socket says 172.21.15.55:42761) 2024-03-29T11:48:43.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.583+0000 7ffbcc973640 1 -- 172.21.15.55:0/2138171684 learned_addr learned my addr 172.21.15.55:0/2138171684 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:43.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 10 client.?.objecter ms_handle_connect 0x7ffbc80ae220 2024-03-29T11:48:43.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:43.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 10 client.?.objecter ms_handle_connect 0x7ffbc80b3f30 2024-03-29T11:48:43.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:43.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395514344 0 0) 0x7ffbc8133d30 con 0x7ffbc80ae220 2024-03-29T11:48:43.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:43.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:43.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:43.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:48:43.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.584+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb4004eb0 con 0x7ffbc80ae220 2024-03-29T11:48:43.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.585+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897612814 0 0) 0x7ffbc813be30 con 0x7ffbc80b3f30 2024-03-29T11:48:43.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.587+0000 7ffbc57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:43.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.587+0000 7ffbc57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:43.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.587+0000 7ffbc57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:43.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.587+0000 7ffbc57fa640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:48:43.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.587+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc8133d30 con 0x7ffbc80b3f30 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.588+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2505087935 0 0) 0x7ffbb4004eb0 con 0x7ffbc80ae220 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.588+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbc813be30 con 0x7ffbc80ae220 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.588+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ffbc0001e10 con 0x7ffbc80ae220 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.588+0000 7ffbc57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.588+0000 7ffbc57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.588+0000 7ffbc57fa640 10 monclient(hunting): dump: 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:43.594 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:43.595 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:43.595 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:43.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.588+0000 7ffbc57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:43.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.588+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 584725708 0 0) 0x7ffbc813be30 con 0x7ffbc80ae220 2024-03-29T11:48:43.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.589+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 >> v1:172.21.15.55:6790/0 conn(0x7ffbc80b3f30 legacy=0x7ffbc8134050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:43.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.590+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 >> v1:172.21.15.171:6789/0 conn(0x7ffbc813b2c0 legacy=0x7ffbc8134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:43.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:48:43.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 1 monclient: found mon.a 2024-03-29T11:48:43.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:43.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:43.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc813bfc0 con 0x7ffbc80ae220 2024-03-29T11:48:43.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 10 monclient: _finish_auth 0 2024-03-29T11:48:43.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:48:43.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:43.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 10 client.?.objecter ms_handle_connect 0x7ffbc813b2c0 2024-03-29T11:48:43.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbc57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:43.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbcd975640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:48:43.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.591+0000 7ffbcd975640 10 monclient: _renew_subs 2024-03-29T11:48:43.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.592+0000 7ffbcd975640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:43.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.592+0000 7ffbcd975640 1 -- 172.21.15.55:0/2138171684 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc8131250 con 0x7ffbc80ae220 2024-03-29T11:48:43.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.592+0000 7ffbcd975640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:43.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.592+0000 7ffbcd975640 10 monclient: _renew_subs 2024-03-29T11:48:43.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.592+0000 7ffbcd975640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:43.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.592+0000 7ffbcd975640 1 -- 172.21.15.55:0/2138171684 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc80b2430 con 0x7ffbc80ae220 2024-03-29T11:48:43.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.592+0000 7ffbcd975640 1 librados: init done 2024-03-29T11:48:43.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.593+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbc0002010 con 0x7ffbc80ae220 2024-03-29T11:48:43.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.593+0000 7ffbc57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.593+0000 7ffbc67fc640 4 set_mon_vals no callback set 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.593+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ffbc0003dc0 con 0x7ffbc80ae220 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.593+0000 7ffbc57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.593+0000 7ffbc57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.593+0000 7ffbc57fa640 10 monclient: dump: 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:43.600 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:43.601 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:43.601 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:43.601 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:43.601 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:43.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.595+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7ffbc0003f20 con 0x7ffbc80ae220 2024-03-29T11:48:43.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.595+0000 7ffbaeffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:43.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.595+0000 7ffbaeffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:43.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.595+0000 7ffbaeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:43.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.595+0000 7ffbaeffd640 1 -- 172.21.15.55:0/2138171684 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc81db740 con 0x7ffbc80ae220 2024-03-29T11:48:43.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.596+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7ffbc0084eb0 con 0x7ffbc80ae220 2024-03-29T11:48:43.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.596+0000 7ffbc57fa640 10 client.4470.objecter ms_dispatch 0x7ffbc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:43.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.596+0000 7ffbc57fa640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:43.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.596+0000 7ffbc7fff640 10 monclient: get_auth_request con 0x7ffbb406db30 auth_method 0 2024-03-29T11:48:43.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.596+0000 7ffbc57fa640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:43.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.596+0000 7ffbc57fa640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-29T11:48:43.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.596+0000 7ffbc57fa640 10 client.4470.objecter ms_handle_connect 0x7ffbb406db30 2024-03-29T11:48:43.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.604+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbc0050790 con 0x7ffbc80ae220 2024-03-29T11:48:43.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.604+0000 7ffbc57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:43.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.604+0000 7ffbc57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:43.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.934+0000 7ffbaeffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:43.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.934+0000 7ffbaeffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:43.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.934+0000 7ffbaeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:43.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.934+0000 7ffbaeffd640 1 -- 172.21.15.55:0/2138171684 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbc804f730 con 0x7ffbc80ae220 2024-03-29T11:48:43.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.936+0000 7ffbc57fa640 1 -- 172.21.15.55:0/2138171684 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7ffbc00526c0 con 0x7ffbc80ae220 2024-03-29T11:48:43.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.936+0000 7ffbc57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:43.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.936+0000 7ffbc57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:43.943 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:48:43.943 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:43.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.942+0000 7ffbaeffd640 10 librados: watch_flush enter 2024-03-29T11:48:43.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.943+0000 7ffbaeffd640 10 librados: watch_flush exit 2024-03-29T11:48:43.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.943+0000 7ffbaeffd640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-29T11:48:43.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.943+0000 7ffbaeffd640 10 client.4470.objecter successfully canceled tick 2024-03-29T11:48:43.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.943+0000 7ffbaeffd640 1 -- 172.21.15.55:0/2138171684 >> v1:172.21.15.171:6816/1463197163 conn(0x7ffbb406db30 legacy=0x7ffbb406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:43.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.943+0000 7ffbaeffd640 10 monclient: shutdown 2024-03-29T11:48:43.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.943+0000 7ffbaeffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:43.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.943+0000 7ffbaeffd640 1 -- 172.21.15.55:0/2138171684 >> v1:172.21.15.55:6789/0 conn(0x7ffbc80ae220 legacy=0x7ffbc8130b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:43.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.943+0000 7ffbaeffd640 1 -- 172.21.15.55:0/2138171684 shutdown_connections 2024-03-29T11:48:43.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.943+0000 7ffbaeffd640 1 -- 172.21.15.55:0/2138171684 >> 172.21.15.55:0/2138171684 conn(0x7ffbc80673e0 msgr2=0x7ffbc80a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:43.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.945+0000 7ffbaeffd640 1 -- 172.21.15.55:0/2138171684 shutdown_connections 2024-03-29T11:48:43.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.945+0000 7ffbaeffd640 1 -- 172.21.15.55:0/2138171684 wait complete. 2024-03-29T11:48:43.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:43.946+0000 7ffbaeffd640 1 librados: shutdown 2024-03-29T11:48:43.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:50.777 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:50.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.922+0000 7f6559990640 10 monclient: get_monmap_and_config 2024-03-29T11:48:50.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.922+0000 7f6559990640 10 monclient: build_initial_monmap 2024-03-29T11:48:50.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.922+0000 7f6559990640 10 monclient: monmap: 2024-03-29T11:48:50.930 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:50.930 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:50.930 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:48:50.923633+0000 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:48:50.923633+0000 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.924+0000 7f6559990640 1 Processor -- start 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.924+0000 7f6559990640 1 -- start start 2024-03-29T11:48:50.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.924+0000 7f6559990640 10 monclient: init 2024-03-29T11:48:50.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.925+0000 7f6559990640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:50.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.925+0000 7f6559990640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:50.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.925+0000 7f6559990640 10 monclient(hunting): picked mon.noname-b con 0x7f65540b4330 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:50.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.925+0000 7f6559990640 10 monclient(hunting): picked mon.noname-c con 0x7f65541470b0 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:50.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6552ffd640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f65540b4330 0x7f65540b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:35818/0 (socket says 172.21.15.55:35818) 2024-03-29T11:48:50.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6552ffd640 1 -- 172.21.15.55:0/3502863681 learned_addr learned my addr 172.21.15.55:0/3502863681 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:50.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.925+0000 7f6559990640 10 monclient(hunting): picked mon.noname-a con 0x7f65541439b0 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6559990640 1 -- 172.21.15.55:0/3502863681 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655406f2a0 con 0x7f65541439b0 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6559990640 1 -- 172.21.15.55:0/3502863681 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6554146b90 con 0x7f65541470b0 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6559990640 1 -- 172.21.15.55:0/3502863681 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540b2ca0 con 0x7f65540b4330 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6559990640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6559990640 10 monclient(hunting): authenticate will time out at 1344.206082s 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165197612 0 0) 0x7f65540b2ca0 con 0x7f65540b4330 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f653c003520 con 0x7f65540b4330 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991383465 0 0) 0x7f655406f2a0 con 0x7f65541439b0 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.926+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65540b2ca0 con 0x7f65541439b0 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3976476582 0 0) 0x7f65540b2ca0 con 0x7f65541439b0 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 >> v1:172.21.15.55:6790/0 conn(0x7f65540b4330 legacy=0x7f65540b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 >> v1:172.21.15.171:6789/0 conn(0x7f65541470b0 legacy=0x7f65540b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 1 monclient: found mon.noname-a 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655406e0c0 con 0x7f65541439b0 2024-03-29T11:48:50.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: _finish_auth 0 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6544001670 con 0x7f65541439b0 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: dump: 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:50.937 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6559990640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6559990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65440017d0 con 0x7f65541439b0 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 1 -- 172.21.15.55:0/3502863681 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6544001c20 con 0x7f65541439b0 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6551ffb640 10 monclient: dump: 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.938 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6559990640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.927+0000 7f6559990640 4 set_mon_vals no callback set 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.928+0000 7f6559990640 10 monclient: shutdown 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.928+0000 7f6559990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.928+0000 7f6559990640 1 -- 172.21.15.55:0/3502863681 >> v1:172.21.15.55:6789/0 conn(0x7f65541439b0 legacy=0x7f6554151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.928+0000 7f6559990640 1 -- 172.21.15.55:0/3502863681 shutdown_connections 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.928+0000 7f6559990640 1 -- 172.21.15.55:0/3502863681 >> 172.21.15.55:0/3502863681 conn(0x7f6554066a00 msgr2=0x7f6554066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.929+0000 7f6559990640 1 -- 172.21.15.55:0/3502863681 shutdown_connections 2024-03-29T11:48:50.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.929+0000 7f6559990640 1 -- 172.21.15.55:0/3502863681 wait complete. 2024-03-29T11:48:50.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.929+0000 7f6559990640 10 monclient: build_initial_monmap 2024-03-29T11:48:50.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.929+0000 7f6559990640 10 monclient: monmap: 2024-03-29T11:48:50.941 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:50.941 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:50.941 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:48:50.941 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:48:50.941 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:50.941 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.930+0000 7f6559990640 1 librados: starting msgr at 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.930+0000 7f6559990640 1 librados: starting objecter 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.930+0000 7f6559990640 1 Processor -- start 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.930+0000 7f6559990640 1 -- start start 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.930+0000 7f6559990640 1 librados: setting wanted keys 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.930+0000 7f6559990640 1 librados: calling monclient init 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.930+0000 7f6559990640 10 monclient: init 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 10 monclient(hunting): picked mon.noname-c con 0x7f65540b4330 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:50.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 10 monclient(hunting): picked mon.noname-b con 0x7f65541439b0 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 10 monclient(hunting): picked mon.noname-a con 0x7f65541470b0 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540ac870 con 0x7f65541470b0 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540acb50 con 0x7f65541439b0 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6554137120 con 0x7f65540b4330 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.931+0000 7f6559990640 10 monclient(hunting): authenticate will time out at 1344.211113s 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f65537fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f65541470b0 0x7f65541358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37104/0 (socket says 172.21.15.55:37104) 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f65537fe640 1 -- 172.21.15.55:0/2046992713 learned_addr learned my addr 172.21.15.55:0/2046992713 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 10 client.?.objecter ms_handle_connect 0x7f65541470b0 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 10 client.?.objecter ms_handle_connect 0x7f65540b4330 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980883443 0 0) 0x7f65540ac870 con 0x7f65541470b0 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.932+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6540004eb0 con 0x7f65541470b0 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3325760898 0 0) 0x7f6540004eb0 con 0x7f65541470b0 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65540ac870 con 0x7f65541470b0 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6544001be0 con 0x7f65541470b0 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): dump: 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:50.944 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347179079 0 0) 0x7f6554137120 con 0x7f65540b4330 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6540004eb0 con 0x7f65540b4330 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 client.?.objecter ms_handle_connect 0x7f65541439b0 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3961229881 0 0) 0x7f6540004eb0 con 0x7f65540b4330 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6554137120 con 0x7f65540b4330 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f654c001bd0 con 0x7f65540b4330 2024-03-29T11:48:50.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.933+0000 7f653bfff640 10 monclient(hunting): dump: 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3410785978 0 0) 0x7f65540ac870 con 0x7f65541470b0 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 >> v1:172.21.15.55:6790/0 conn(0x7f65540b4330 legacy=0x7f65540aa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 >> v1:172.21.15.171:6789/0 conn(0x7f65541439b0 legacy=0x7f65540a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 1 monclient: found mon.a 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65540aa960 con 0x7f65541470b0 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 10 monclient: _finish_auth 0 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f653bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f6559990640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f6559990640 10 monclient: _renew_subs 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f6559990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65540a9bf0 con 0x7f65541470b0 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f6559990640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f6559990640 10 monclient: _renew_subs 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f6559990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:50.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6554137820 con 0x7f65541470b0 2024-03-29T11:48:50.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.934+0000 7f6559990640 1 librados: init done 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.935+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6544003910 con 0x7f65541470b0 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.935+0000 7f653bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.935+0000 7f65517fa640 4 set_mon_vals no callback set 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.936+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f6544003ca0 con 0x7f65541470b0 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.936+0000 7f653bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.936+0000 7f653bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.936+0000 7f653bfff640 10 monclient: dump: 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:50.949 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.936+0000 7f6559990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.936+0000 7f6559990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.936+0000 7f6559990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.936+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6554136780 con 0x7f65541470b0 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.937+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f6544003e00 con 0x7f65541470b0 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.938+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f6544084bc0 con 0x7f65541470b0 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.938+0000 7f653bfff640 10 client.4476.objecter ms_dispatch 0x7f6554000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.938+0000 7f653bfff640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.938+0000 7f653bfff640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.938+0000 7f653bfff640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T11:48:50.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.939+0000 7f6552ffd640 10 monclient: get_auth_request con 0x7f654006ded0 auth_method 0 2024-03-29T11:48:50.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.940+0000 7f653bfff640 10 client.4476.objecter ms_handle_connect 0x7f654006ded0 2024-03-29T11:48:50.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.946+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6544050500 con 0x7f65541470b0 2024-03-29T11:48:50.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.946+0000 7f653bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:50.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:50.946+0000 7f653bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:51.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.270+0000 7f6559990640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:51.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.270+0000 7f6559990640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:51.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.270+0000 7f6559990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:51.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.270+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6554075f10 con 0x7f65541470b0 2024-03-29T11:48:51.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.271+0000 7f653bfff640 1 -- 172.21.15.55:0/2046992713 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f6544084270 con 0x7f65541470b0 2024-03-29T11:48:51.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.271+0000 7f653bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:51.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.271+0000 7f653bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:51.277 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:48:51.277 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:51.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.274+0000 7f6559990640 10 librados: watch_flush enter 2024-03-29T11:48:51.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.274+0000 7f6559990640 10 librados: watch_flush exit 2024-03-29T11:48:51.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.274+0000 7f6559990640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T11:48:51.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.274+0000 7f6559990640 10 client.4476.objecter successfully canceled tick 2024-03-29T11:48:51.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.274+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 >> v1:172.21.15.171:6816/1463197163 conn(0x7f654006ded0 legacy=0x7f6540070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:51.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.274+0000 7f6559990640 10 monclient: shutdown 2024-03-29T11:48:51.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.275+0000 7f6559990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:51.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.275+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 >> v1:172.21.15.55:6789/0 conn(0x7f65541470b0 legacy=0x7f65541358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:51.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.275+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 shutdown_connections 2024-03-29T11:48:51.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.275+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 >> 172.21.15.55:0/2046992713 conn(0x7f6554066a00 msgr2=0x7f6554070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:51.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.275+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 shutdown_connections 2024-03-29T11:48:51.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.275+0000 7f6559990640 1 -- 172.21.15.55:0/2046992713 wait complete. 2024-03-29T11:48:51.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:51.275+0000 7f6559990640 1 librados: shutdown 2024-03-29T11:48:51.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:58.103 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:58.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.248+0000 7fa3c9472640 10 monclient: get_monmap_and_config 2024-03-29T11:48:58.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.249+0000 7fa3c9472640 10 monclient: build_initial_monmap 2024-03-29T11:48:58.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.249+0000 7fa3c9472640 10 monclient: monmap: 2024-03-29T11:48:58.255 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:58.255 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:58.255 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:48:58.250201+0000 2024-03-29T11:48:58.255 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:48:58.250201+0000 2024-03-29T11:48:58.255 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:58.256 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:58.256 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:58.256 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:48:58.256 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:48:58.256 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:58.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.250+0000 7fa3c9472640 1 Processor -- start 2024-03-29T11:48:58.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.250+0000 7fa3c9472640 1 -- start start 2024-03-29T11:48:58.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.250+0000 7fa3c9472640 10 monclient: init 2024-03-29T11:48:58.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.251+0000 7fa3c9472640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:58.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.251+0000 7fa3c9472640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:48:58.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c9472640 10 monclient(hunting): picked mon.noname-a con 0x7fa3bc006eb0 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:58.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c9472640 10 monclient(hunting): picked mon.noname-c con 0x7fa3bc038f10 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:58.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c8c71640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fa3bc036690 0x7fa3bc036b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:46862/0 (socket says 172.21.15.55:46862) 2024-03-29T11:48:58.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c8c71640 1 -- 172.21.15.55:0/3147995432 learned_addr learned my addr 172.21.15.55:0/3147995432 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:58.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c9472640 10 monclient(hunting): picked mon.noname-b con 0x7fa3bc036690 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:58.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c9472640 1 -- 172.21.15.55:0/3147995432 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3bc033c90 con 0x7fa3bc006eb0 2024-03-29T11:48:58.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c9472640 1 -- 172.21.15.55:0/3147995432 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3bc0389f0 con 0x7fa3bc038f10 2024-03-29T11:48:58.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c9472640 1 -- 172.21.15.55:0/3147995432 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3bc030890 con 0x7fa3bc036690 2024-03-29T11:48:58.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c9472640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:58.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.253+0000 7fa3c9472640 10 monclient(hunting): authenticate will time out at 1351.533125s 2024-03-29T11:48:58.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594124578 0 0) 0x7fa3bc030890 con 0x7fa3bc036690 2024-03-29T11:48:58.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:58.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:58.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:58.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:48:58.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b4003520 con 0x7fa3bc036690 2024-03-29T11:48:58.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561409459 0 0) 0x7fa3bc033c90 con 0x7fa3bc006eb0 2024-03-29T11:48:58.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:58.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:58.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:58.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:48:58.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3bc030890 con 0x7fa3bc006eb0 2024-03-29T11:48:58.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3205155678 0 0) 0x7fa3b4003520 con 0x7fa3bc036690 2024-03-29T11:48:58.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.255+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 >> v1:172.21.15.171:6789/0 conn(0x7fa3bc038f10 legacy=0x7fa3bc039330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 >> v1:172.21.15.55:6789/0 conn(0x7fa3bc006eb0 legacy=0x7fa3bc031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3bc0cbc40 con 0x7fa3bc036690 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa3c404f730 con 0x7fa3bc036690 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:58.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c2ffd640 10 monclient: dump: 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c9472640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.257+0000 7fa3c9472640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:58.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.258+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3c40008d0 con 0x7fa3bc036690 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.258+0000 7fa3c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.258+0000 7fa3c2ffd640 1 -- 172.21.15.55:0/3147995432 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa3c40758b0 con 0x7fa3bc036690 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.258+0000 7fa3c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.258+0000 7fa3c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.258+0000 7fa3c2ffd640 10 monclient: dump: 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:58.266 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:58.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.258+0000 7fa3c9472640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:58.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.258+0000 7fa3c9472640 4 set_mon_vals no callback set 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.259+0000 7fa3c9472640 10 monclient: shutdown 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.259+0000 7fa3c9472640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.259+0000 7fa3c9472640 1 -- 172.21.15.55:0/3147995432 >> v1:172.21.15.55:6790/0 conn(0x7fa3bc036690 legacy=0x7fa3bc036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.259+0000 7fa3c9472640 1 -- 172.21.15.55:0/3147995432 shutdown_connections 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.259+0000 7fa3c9472640 1 -- 172.21.15.55:0/3147995432 >> 172.21.15.55:0/3147995432 conn(0x7fa3bc005c20 msgr2=0x7fa3bc006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.260+0000 7fa3c9472640 1 -- 172.21.15.55:0/3147995432 shutdown_connections 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.261+0000 7fa3c9472640 1 -- 172.21.15.55:0/3147995432 wait complete. 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.261+0000 7fa3c9472640 10 monclient: build_initial_monmap 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.261+0000 7fa3c9472640 10 monclient: monmap: 2024-03-29T11:48:58.269 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.262+0000 7fa3c9472640 1 librados: starting msgr at 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.262+0000 7fa3c9472640 1 librados: starting objecter 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.262+0000 7fa3c9472640 1 Processor -- start 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.263+0000 7fa3c9472640 1 -- start start 2024-03-29T11:48:58.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.263+0000 7fa3c9472640 1 librados: setting wanted keys 2024-03-29T11:48:58.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.263+0000 7fa3c9472640 1 librados: calling monclient init 2024-03-29T11:48:58.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.263+0000 7fa3c9472640 10 monclient: init 2024-03-29T11:48:58.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.263+0000 7fa3c9472640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:58.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.264+0000 7fa3c9472640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:48:58.272 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.266+0000 7fa3c9472640 10 monclient(hunting): picked mon.noname-a con 0x7fa3bc006eb0 addr v1:172.21.15.55:6789/0 2024-03-29T11:48:58.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.266+0000 7fa3c9472640 10 monclient(hunting): picked mon.noname-b con 0x7fa3bc036690 addr v1:172.21.15.171:6789/0 2024-03-29T11:48:58.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.266+0000 7fa3c9472640 10 monclient(hunting): picked mon.noname-c con 0x7fa3bc038f10 addr v1:172.21.15.55:6790/0 2024-03-29T11:48:58.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.266+0000 7fa3c9472640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3bc00d330 con 0x7fa3bc006eb0 2024-03-29T11:48:58.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.266+0000 7fa3c9472640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3bc165160 con 0x7fa3bc036690 2024-03-29T11:48:58.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c9472640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3bc02d3d0 con 0x7fa3bc038f10 2024-03-29T11:48:58.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c9472640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:58.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c9472640 10 monclient(hunting): authenticate will time out at 1351.546664s 2024-03-29T11:48:58.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c3fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa3bc006eb0 0x7fa3bc00bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:38935/0 (socket says 172.21.15.55:38935) 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c3fff640 1 -- 172.21.15.55:0/2931520995 learned_addr learned my addr 172.21.15.55:0/2931520995 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 10 client.?.objecter ms_handle_connect 0x7fa3bc006eb0 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 10 client.?.objecter ms_handle_connect 0x7fa3bc038f10 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045092427 0 0) 0x7fa3bc00d330 con 0x7fa3bc006eb0 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:48:58.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.267+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a8004eb0 con 0x7fa3bc006eb0 2024-03-29T11:48:58.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274982674 0 0) 0x7fa3bc02d3d0 con 0x7fa3bc038f10 2024-03-29T11:48:58.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:58.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:58.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:58.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:48:58.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3bc00d330 con 0x7fa3bc038f10 2024-03-29T11:48:58.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1968648027 0 0) 0x7fa3a8004eb0 con 0x7fa3bc006eb0 2024-03-29T11:48:58.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3bc02d3d0 con 0x7fa3bc006eb0 2024-03-29T11:48:58.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa3b8001a70 con 0x7fa3bc006eb0 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:58.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2531120513 0 0) 0x7fa3bc00d330 con 0x7fa3bc038f10 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3a8004eb0 con 0x7fa3bc038f10 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa3c404f400 con 0x7fa3bc038f10 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:58.277 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:58.278 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:58.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2910632691 0 0) 0x7fa3bc02d3d0 con 0x7fa3bc006eb0 2024-03-29T11:48:58.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 >> v1:172.21.15.55:6790/0 conn(0x7fa3bc038f10 legacy=0x7fa3bc164a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:58.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.268+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 >> v1:172.21.15.171:6789/0 conn(0x7fa3bc036690 legacy=0x7fa3bc164340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:48:58.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:48:58.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 1 monclient: found mon.a 2024-03-29T11:48:58.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:58.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:58.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3bc00b4e0 con 0x7fa3bc006eb0 2024-03-29T11:48:58.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:48:58.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:48:58.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:58.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3b8001bd0 con 0x7fa3bc006eb0 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.270+0000 7fa3c27fc640 4 set_mon_vals no callback set 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.271+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa3b8003a40 con 0x7fa3bc006eb0 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.271+0000 7fa3c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.271+0000 7fa3c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.271+0000 7fa3c0ff9640 10 monclient: dump: 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:48:58.281 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.272+0000 7fa3c9472640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.272+0000 7fa3c9472640 10 monclient: _renew_subs 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.272+0000 7fa3c9472640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.272+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3bc00b7c0 con 0x7fa3bc006eb0 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.272+0000 7fa3c9472640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.272+0000 7fa3c9472640 10 monclient: _renew_subs 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.272+0000 7fa3c9472640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.272+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3bc00d650 con 0x7fa3bc006eb0 2024-03-29T11:48:58.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.272+0000 7fa3c9472640 1 librados: init done 2024-03-29T11:48:58.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.275+0000 7fa3c9472640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:58.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.275+0000 7fa3c9472640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:58.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.275+0000 7fa3c9472640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:58.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.275+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3bc007b70 con 0x7fa3bc006eb0 2024-03-29T11:48:58.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.275+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fa3b8003ba0 con 0x7fa3bc006eb0 2024-03-29T11:48:58.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.276+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fa3b8084b20 con 0x7fa3bc006eb0 2024-03-29T11:48:58.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.276+0000 7fa3c0ff9640 10 client.4482.objecter ms_dispatch 0x7fa3c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:58.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.276+0000 7fa3c0ff9640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:58.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.276+0000 7fa3c0ff9640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:58.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.276+0000 7fa3c0ff9640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T11:48:58.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.277+0000 7fa3c37fe640 10 monclient: get_auth_request con 0x7fa3a806de50 auth_method 0 2024-03-29T11:48:58.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.277+0000 7fa3c0ff9640 10 client.4482.objecter ms_handle_connect 0x7fa3a806de50 2024-03-29T11:48:58.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.284+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3b8050400 con 0x7fa3bc006eb0 2024-03-29T11:48:58.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.284+0000 7fa3c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:58.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.284+0000 7fa3c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:58.616 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:48:58.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.608+0000 7fa3c9472640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:58.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.608+0000 7fa3c9472640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:58.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.608+0000 7fa3c9472640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:48:58.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.608+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3bc004c50 con 0x7fa3bc006eb0 2024-03-29T11:48:58.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.609+0000 7fa3c0ff9640 1 -- 172.21.15.55:0/2931520995 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fa3b8052330 con 0x7fa3bc006eb0 2024-03-29T11:48:58.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.609+0000 7fa3c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:58.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.609+0000 7fa3c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:58.619 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:58.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.613+0000 7fa3c9472640 10 librados: watch_flush enter 2024-03-29T11:48:58.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 10 librados: watch_flush exit 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 10 client.4482.objecter successfully canceled tick 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 >> v1:172.21.15.171:6816/1463197163 conn(0x7fa3a806de50 legacy=0x7fa3a8070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 10 monclient: shutdown 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 >> v1:172.21.15.55:6789/0 conn(0x7fa3bc006eb0 legacy=0x7fa3bc00bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 shutdown_connections 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 >> 172.21.15.55:0/2931520995 conn(0x7fa3bc005c20 msgr2=0x7fa3bc0050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 shutdown_connections 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.616+0000 7fa3c9472640 1 -- 172.21.15.55:0/2931520995 wait complete. 2024-03-29T11:48:58.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:48:58.617+0000 7fa3c9472640 1 librados: shutdown 2024-03-29T11:48:58.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:05.448 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:05.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.590+0000 7f1d55063640 10 monclient: get_monmap_and_config 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.590+0000 7f1d55063640 10 monclient: build_initial_monmap 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.590+0000 7f1d55063640 10 monclient: monmap: 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:49:05.591674+0000 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:49:05.591674+0000 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:49:05.597 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:05.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.591+0000 7f1d55063640 1 Processor -- start 2024-03-29T11:49:05.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.591+0000 7f1d55063640 1 -- start start 2024-03-29T11:49:05.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.591+0000 7f1d55063640 10 monclient: init 2024-03-29T11:49:05.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:05.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:49:05.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 10 monclient(hunting): picked mon.noname-a con 0x7f1d501457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:05.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5013f650 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:05.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 10 monclient(hunting): picked mon.noname-b con 0x7f1d500a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:05.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d500aaef0 con 0x7f1d501457f0 2024-03-29T11:49:05.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d50145bd0 con 0x7f1d5013f650 2024-03-29T11:49:05.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d50145eb0 con 0x7f1d500a9d70 2024-03-29T11:49:05.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:05.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.592+0000 7f1d55063640 10 monclient(hunting): authenticate will time out at 1358.871979s 2024-03-29T11:49:05.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.594+0000 7f1d4f59e640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f1d500a9d70 0x7f1d5014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:43736/0 (socket says 172.21.15.55:43736) 2024-03-29T11:49:05.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.594+0000 7f1d4f59e640 1 -- 172.21.15.55:0/555290569 learned_addr learned my addr 172.21.15.55:0/555290569 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:05.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.595+0000 7f1d4dd9b640 1 -- 172.21.15.55:0/555290569 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956091291 0 0) 0x7f1d50145eb0 con 0x7f1d500a9d70 2024-03-29T11:49:05.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.595+0000 7f1d4dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:49:05.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.595+0000 7f1d4dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:05.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.595+0000 7f1d4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:05.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.595+0000 7f1d4dd9b640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:49:05.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.595+0000 7f1d4dd9b640 1 -- 172.21.15.55:0/555290569 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d38003520 con 0x7f1d500a9d70 2024-03-29T11:49:05.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.595+0000 7f1d4dd9b640 1 -- 172.21.15.55:0/555290569 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177670353 0 0) 0x7f1d38003520 con 0x7f1d500a9d70 2024-03-29T11:49:05.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.595+0000 7f1d4dd9b640 1 -- 172.21.15.55:0/555290569 >> v1:172.21.15.171:6789/0 conn(0x7f1d5013f650 legacy=0x7f1d500b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:49:05.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 1 -- 172.21.15.55:0/555290569 >> v1:172.21.15.55:6789/0 conn(0x7f1d501457f0 legacy=0x7f1d5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 1 monclient: found mon.noname-b 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 1 -- 172.21.15.55:0/555290569 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d50146340 con 0x7f1d500a9d70 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d55063640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d55063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 1 -- 172.21.15.55:0/555290569 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1d44001690 con 0x7f1d500a9d70 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:05.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d4dd9b640 10 monclient: dump: 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:05.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.596+0000 7f1d55063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:05.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.599+0000 7f1d4dd9b640 1 -- 172.21.15.55:0/555290569 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d44001920 con 0x7f1d500a9d70 2024-03-29T11:49:05.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.599+0000 7f1d4dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:05.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.599+0000 7f1d55063640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:05.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.599+0000 7f1d55063640 4 set_mon_vals no callback set 2024-03-29T11:49:05.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.600+0000 7f1d55063640 10 monclient: shutdown 2024-03-29T11:49:05.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.600+0000 7f1d55063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:05.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.600+0000 7f1d55063640 1 -- 172.21.15.55:0/555290569 >> v1:172.21.15.55:6790/0 conn(0x7f1d500a9d70 legacy=0x7f1d5014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:05.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.600+0000 7f1d55063640 1 -- 172.21.15.55:0/555290569 shutdown_connections 2024-03-29T11:49:05.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.601+0000 7f1d55063640 1 -- 172.21.15.55:0/555290569 >> 172.21.15.55:0/555290569 conn(0x7f1d50066a00 msgr2=0x7f1d50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:05.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.601+0000 7f1d55063640 1 -- 172.21.15.55:0/555290569 shutdown_connections 2024-03-29T11:49:05.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.601+0000 7f1d55063640 1 -- 172.21.15.55:0/555290569 wait complete. 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.602+0000 7f1d55063640 10 monclient: build_initial_monmap 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.602+0000 7f1d55063640 10 monclient: monmap: 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:05.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.603+0000 7f1d55063640 1 librados: starting msgr at 2024-03-29T11:49:05.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.603+0000 7f1d55063640 1 librados: starting objecter 2024-03-29T11:49:05.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.603+0000 7f1d55063640 1 Processor -- start 2024-03-29T11:49:05.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.604+0000 7f1d55063640 1 -- start start 2024-03-29T11:49:05.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.604+0000 7f1d55063640 1 librados: setting wanted keys 2024-03-29T11:49:05.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.604+0000 7f1d55063640 1 librados: calling monclient init 2024-03-29T11:49:05.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.604+0000 7f1d55063640 10 monclient: init 2024-03-29T11:49:05.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:05.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:49:05.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 10 monclient(hunting): picked mon.noname-a con 0x7f1d5013f650 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:05.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 10 monclient(hunting): picked mon.noname-c con 0x7f1d501457f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:05.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5013fda0 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:05.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d501e3c80 con 0x7f1d5013f650 2024-03-29T11:49:05.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d501e1670 con 0x7f1d5013fda0 2024-03-29T11:49:05.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5006f6d0 con 0x7f1d501457f0 2024-03-29T11:49:05.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:05.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.605+0000 7f1d55063640 10 monclient(hunting): authenticate will time out at 1358.885422s 2024-03-29T11:49:05.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.608+0000 7f1d4ed9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1d5013f650 0x7f1d501e4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41531/0 (socket says 172.21.15.55:41531) 2024-03-29T11:49:05.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.608+0000 7f1d4ed9d640 1 -- 172.21.15.55:0/3795626664 learned_addr learned my addr 172.21.15.55:0/3795626664 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:05.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 client.?.objecter ms_handle_connect 0x7f1d5013f650 2024-03-29T11:49:05.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:05.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 client.?.objecter ms_handle_connect 0x7f1d5013fda0 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 client.?.objecter ms_handle_connect 0x7f1d501457f0 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461084359 0 0) 0x7f1d501e3c80 con 0x7f1d5013f650 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d3c004eb0 con 0x7f1d5013f650 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179855763 0 0) 0x7f1d501e1670 con 0x7f1d5013fda0 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d501e3c80 con 0x7f1d5013fda0 2024-03-29T11:49:05.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923048037 0 0) 0x7f1d5006f6d0 con 0x7f1d501457f0 2024-03-29T11:49:05.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:49:05.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:05.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:05.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:49:05.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.609+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d501e1670 con 0x7f1d501457f0 2024-03-29T11:49:05.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 304540669 0 0) 0x7f1d3c004eb0 con 0x7f1d5013f650 2024-03-29T11:49:05.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d5006f6d0 con 0x7f1d5013f650 2024-03-29T11:49:05.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1d400018f0 con 0x7f1d5013f650 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient(hunting): dump: 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:05.622 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1437501754 0 0) 0x7f1d501e3c80 con 0x7f1d5013fda0 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d3c004eb0 con 0x7f1d5013fda0 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1d440047e0 con 0x7f1d5013fda0 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient(hunting): dump: 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:05.623 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3657873119 0 0) 0x7f1d5006f6d0 con 0x7f1d5013f650 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 >> v1:172.21.15.55:6790/0 conn(0x7f1d501457f0 legacy=0x7f1d501e0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 >> v1:172.21.15.171:6789/0 conn(0x7f1d5013fda0 legacy=0x7f1d501e0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 monclient: found mon.a 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:49:05.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5006f9b0 con 0x7f1d5013f650 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient: _finish_auth 0 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d55063640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d55063640 10 monclient: _renew_subs 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d55063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d50142d70 con 0x7f1d5013f650 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d55063640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d55063640 10 monclient: _renew_subs 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d55063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d50142440 con 0x7f1d5013f650 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.610+0000 7f1d55063640 1 librados: init done 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.612+0000 7f1d55063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:05.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.612+0000 7f1d55063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.612+0000 7f1d55063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.612+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d50075f10 con 0x7f1d5013f650 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.613+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d40003610 con 0x7f1d5013f650 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.613+0000 7f1d377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.613+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1d40003a20 con 0x7f1d5013f650 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.613+0000 7f1d377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.613+0000 7f1d4d59a640 4 set_mon_vals no callback set 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.613+0000 7f1d377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.613+0000 7f1d377fe640 10 monclient: dump: 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:05.626 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.614+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f1d40003c40 con 0x7f1d5013f650 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.615+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f1d400849d0 con 0x7f1d5013f650 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.615+0000 7f1d377fe640 10 client.4488.objecter ms_dispatch 0x7f1d50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.615+0000 7f1d377fe640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.615+0000 7f1d377fe640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:05.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.615+0000 7f1d377fe640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T11:49:05.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.616+0000 7f1d4e59c640 10 monclient: get_auth_request con 0x7f1d3c06e000 auth_method 0 2024-03-29T11:49:05.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.617+0000 7f1d377fe640 10 client.4488.objecter ms_handle_connect 0x7f1d3c06e000 2024-03-29T11:49:05.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.628+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d400502b0 con 0x7f1d5013f650 2024-03-29T11:49:05.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.628+0000 7f1d377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:05.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.628+0000 7f1d377fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:05.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.952+0000 7f1d55063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:05.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.952+0000 7f1d55063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:05.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.952+0000 7f1d55063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:49:05.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.952+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d501e3a70 con 0x7f1d5013f650 2024-03-29T11:49:05.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.954+0000 7f1d377fe640 1 -- 172.21.15.55:0/3795626664 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f1d400521e0 con 0x7f1d5013f650 2024-03-29T11:49:05.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.954+0000 7f1d377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:05.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.954+0000 7f1d377fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:05.960 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:49:05.961 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:05.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.956+0000 7f1d55063640 10 librados: watch_flush enter 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.959+0000 7f1d55063640 10 librados: watch_flush exit 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.959+0000 7f1d55063640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.959+0000 7f1d55063640 10 client.4488.objecter successfully canceled tick 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.959+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 >> v1:172.21.15.171:6816/1463197163 conn(0x7f1d3c06e000 legacy=0x7f1d3c0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.959+0000 7f1d55063640 10 monclient: shutdown 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.959+0000 7f1d55063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.959+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 >> v1:172.21.15.55:6789/0 conn(0x7f1d5013f650 legacy=0x7f1d501e4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.959+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 shutdown_connections 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.959+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 >> 172.21.15.55:0/3795626664 conn(0x7f1d50066a00 msgr2=0x7f1d5013b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.960+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 shutdown_connections 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.960+0000 7f1d55063640 1 -- 172.21.15.55:0/3795626664 wait complete. 2024-03-29T11:49:05.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:05.960+0000 7f1d55063640 1 librados: shutdown 2024-03-29T11:49:05.982 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:12.789 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:12.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.940+0000 7f747b789640 10 monclient: get_monmap_and_config 2024-03-29T11:49:12.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.940+0000 7f747b789640 10 monclient: build_initial_monmap 2024-03-29T11:49:12.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.940+0000 7f747b789640 10 monclient: monmap: 2024-03-29T11:49:12.947 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:12.947 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:49:12.941429+0000 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:49:12.941429+0000 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.941+0000 7f747b789640 1 Processor -- start 2024-03-29T11:49:12.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.941+0000 7f747b789640 1 -- start start 2024-03-29T11:49:12.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.941+0000 7f747b789640 10 monclient: init 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 10 monclient(hunting): picked mon.noname-c con 0x7f74740a9ac0 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 10 monclient(hunting): picked mon.noname-b con 0x7f747413ee90 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 10 monclient(hunting): picked mon.noname-a con 0x7f74740af900 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74740afd10 con 0x7f74740af900 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74740af6f0 con 0x7f74740a9ac0 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74740aac80 con 0x7f747413ee90 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.942+0000 7f747b789640 10 monclient(hunting): authenticate will time out at 1366.222159s 2024-03-29T11:49:12.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.945+0000 7f7478d24640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f747413ee90 0x7f747413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:34029/0 (socket says 172.21.15.55:34029) 2024-03-29T11:49:12.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.945+0000 7f7478d24640 1 -- 172.21.15.55:0/76618382 learned_addr learned my addr 172.21.15.55:0/76618382 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:12.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.947+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024554789 0 0) 0x7f74740aac80 con 0x7f747413ee90 2024-03-29T11:49:12.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.947+0000 7f7467fff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:12.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.947+0000 7f7467fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:12.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.947+0000 7f7467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:12.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.947+0000 7f7467fff640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:49:12.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.947+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460003520 con 0x7f747413ee90 2024-03-29T11:49:12.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.948+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193274257 0 0) 0x7f74740af6f0 con 0x7f74740a9ac0 2024-03-29T11:49:12.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.948+0000 7f7467fff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:12.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.948+0000 7f7467fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:12.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.948+0000 7f7467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:12.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.948+0000 7f7467fff640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:49:12.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.948+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74740aac80 con 0x7f74740a9ac0 2024-03-29T11:49:12.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.948+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1983519833 0 0) 0x7f7460003520 con 0x7f747413ee90 2024-03-29T11:49:12.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.948+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 >> v1:172.21.15.171:6789/0 conn(0x7f74740a9ac0 legacy=0x7f74740adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.948+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 >> v1:172.21.15.55:6789/0 conn(0x7f74740af900 legacy=0x7f7474149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:49:12.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 10 monclient: _finish_hunting 0 2024-03-29T11:49:12.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 1 monclient: found mon.noname-b 2024-03-29T11:49:12.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:12.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74740aaf60 con 0x7f747413ee90 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 10 monclient: _finish_auth 0 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 10 monclient: _check_auth_tickets 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f746c004c90 con 0x7f747413ee90 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.949+0000 7f7467fff640 10 monclient: dump: 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:12.957 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:12.958 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:12.958 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:12.958 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:12.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.951+0000 7f747b789640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:49:12.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.951+0000 7f747b789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:12.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.953+0000 7f7467fff640 1 -- 172.21.15.55:0/76618382 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f746c004df0 con 0x7f747413ee90 2024-03-29T11:49:12.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.953+0000 7f7467fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:12.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.953+0000 7f747b789640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:12.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.953+0000 7f747b789640 4 set_mon_vals no callback set 2024-03-29T11:49:12.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.954+0000 7f747b789640 10 monclient: shutdown 2024-03-29T11:49:12.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.954+0000 7f747b789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:12.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.954+0000 7f747b789640 1 -- 172.21.15.55:0/76618382 >> v1:172.21.15.55:6790/0 conn(0x7f747413ee90 legacy=0x7f747413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.954+0000 7f747b789640 1 -- 172.21.15.55:0/76618382 shutdown_connections 2024-03-29T11:49:12.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.954+0000 7f747b789640 1 -- 172.21.15.55:0/76618382 >> 172.21.15.55:0/76618382 conn(0x7f74740b35e0 msgr2=0x7f74740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:12.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.954+0000 7f747b789640 1 -- 172.21.15.55:0/76618382 shutdown_connections 2024-03-29T11:49:12.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.955+0000 7f747b789640 1 -- 172.21.15.55:0/76618382 wait complete. 2024-03-29T11:49:12.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.955+0000 7f747b789640 10 monclient: build_initial_monmap 2024-03-29T11:49:12.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.955+0000 7f747b789640 10 monclient: monmap: 2024-03-29T11:49:12.965 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.956+0000 7f747b789640 1 librados: starting msgr at 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.956+0000 7f747b789640 1 librados: starting objecter 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.956+0000 7f747b789640 1 Processor -- start 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.956+0000 7f747b789640 1 -- start start 2024-03-29T11:49:12.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.956+0000 7f747b789640 1 librados: setting wanted keys 2024-03-29T11:49:12.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.956+0000 7f747b789640 1 librados: calling monclient init 2024-03-29T11:49:12.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.956+0000 7f747b789640 10 monclient: init 2024-03-29T11:49:12.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.957+0000 7f747b789640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:12.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.957+0000 7f747b789640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:49:12.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.957+0000 7f747b789640 10 monclient(hunting): picked mon.noname-a con 0x7f74740a9ac0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:12.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.957+0000 7f747b789640 10 monclient(hunting): picked mon.noname-b con 0x7f74740af900 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:12.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.960+0000 7f747b789640 10 monclient(hunting): picked mon.noname-c con 0x7f747413ee90 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:12.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.960+0000 7f747b789640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747406f530 con 0x7f74740a9ac0 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.960+0000 7f747b789640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7474138d60 con 0x7f74740af900 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.960+0000 7f747b789640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7474070860 con 0x7f747413ee90 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.960+0000 7f747b789640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.960+0000 7f747b789640 10 monclient(hunting): authenticate will time out at 1366.239939s 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.960+0000 7f7479d26640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f747413ee90 0x7f7474138650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:34129/0 (socket says 172.21.15.55:34129) 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.960+0000 7f7479d26640 1 -- 172.21.15.55:0/1467891407 learned_addr learned my addr 172.21.15.55:0/1467891407 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.961+0000 7f7465ffb640 10 client.?.objecter ms_handle_connect 0x7f747413ee90 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.961+0000 7f7465ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.961+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111464566 0 0) 0x7f7474070860 con 0x7f747413ee90 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.961+0000 7f7465ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.961+0000 7f7465ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.961+0000 7f7465ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:12.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.961+0000 7f7465ffb640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.961+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7458004eb0 con 0x7f747413ee90 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1559633284 0 0) 0x7f7458004eb0 con 0x7f747413ee90 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7474070860 con 0x7f747413ee90 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f74680018e0 con 0x7f747413ee90 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 monclient(hunting): dump: 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:12.971 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2866092885 0 0) 0x7f7474070860 con 0x7f747413ee90 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 >> v1:172.21.15.171:6789/0 conn(0x7f74740af900 legacy=0x7f7474137f40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 >> v1:172.21.15.55:6789/0 conn(0x7f74740a9ac0 legacy=0x7f74741dba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 1 monclient: found mon.c 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747406f010 con 0x7f747413ee90 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 monclient: _finish_auth 0 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 client.?.objecter ms_handle_connect 0x7f74740a9ac0 2024-03-29T11:49:12.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f7465ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:12.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f747b789640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:49:12.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f747b789640 10 monclient: _renew_subs 2024-03-29T11:49:12.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f747b789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:12.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.962+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7474072560 con 0x7f747413ee90 2024-03-29T11:49:12.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.963+0000 7f747b789640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:12.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.963+0000 7f747b789640 10 monclient: _renew_subs 2024-03-29T11:49:12.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.963+0000 7f747b789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:12.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.963+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f747406ebf0 con 0x7f747413ee90 2024-03-29T11:49:12.973 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.963+0000 7f747b789640 1 librados: init done 2024-03-29T11:49:12.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.966+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74680035f0 con 0x7f747413ee90 2024-03-29T11:49:12.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.966+0000 7f7465ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:12.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.966+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7468003a00 con 0x7f747413ee90 2024-03-29T11:49:12.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.966+0000 7f74677fe640 4 set_mon_vals no callback set 2024-03-29T11:49:12.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.967+0000 7f747b789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:12.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.967+0000 7f747b789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:12.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.967+0000 7f747b789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:12.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.967+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7454005080 con 0x7f747413ee90 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.968+0000 7f7465ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.968+0000 7f7465ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.968+0000 7f7465ffb640 10 monclient: dump: 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.968+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f7468003c20 con 0x7f747413ee90 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.969+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f7468084b80 con 0x7f747413ee90 2024-03-29T11:49:12.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.969+0000 7f7465ffb640 10 client.4493.objecter ms_dispatch 0x7f7474000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:12.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.969+0000 7f7465ffb640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:12.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.969+0000 7f7465ffb640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:12.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.969+0000 7f7465ffb640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:49:12.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.970+0000 7f7479525640 10 monclient: get_auth_request con 0x7f745806dc70 auth_method 0 2024-03-29T11:49:12.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.971+0000 7f7465ffb640 10 client.4493.objecter ms_handle_connect 0x7f745806dc70 2024-03-29T11:49:12.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.976+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74680504e0 con 0x7f747413ee90 2024-03-29T11:49:12.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.976+0000 7f7465ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:12.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:12.976+0000 7f7465ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:13.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.313+0000 7f747b789640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:13.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.313+0000 7f747b789640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:13.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.313+0000 7f747b789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:13.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.313+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7454005330 con 0x7f747413ee90 2024-03-29T11:49:13.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.315+0000 7f7465ffb640 1 -- 172.21.15.55:0/1467891407 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f7468052410 con 0x7f747413ee90 2024-03-29T11:49:13.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.315+0000 7f7465ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:13.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.315+0000 7f7465ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:13.322 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:49:13.322 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.319+0000 7f747b789640 10 librados: watch_flush enter 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.319+0000 7f747b789640 10 librados: watch_flush exit 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.319+0000 7f747b789640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.319+0000 7f747b789640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.320+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 >> v1:172.21.15.171:6816/1463197163 conn(0x7f745806dc70 legacy=0x7f7458070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.320+0000 7f747b789640 10 monclient: shutdown 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.320+0000 7f747b789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.320+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 >> v1:172.21.15.55:6790/0 conn(0x7f747413ee90 legacy=0x7f7474138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.320+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 shutdown_connections 2024-03-29T11:49:13.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.320+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 >> 172.21.15.55:0/1467891407 conn(0x7f74740b35e0 msgr2=0x7f747413b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:13.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.322+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 shutdown_connections 2024-03-29T11:49:13.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.322+0000 7f747b789640 1 -- 172.21.15.55:0/1467891407 wait complete. 2024-03-29T11:49:13.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:13.322+0000 7f747b789640 1 librados: shutdown 2024-03-29T11:49:13.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:20.152 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:20.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.294+0000 7f4efea67640 10 monclient: get_monmap_and_config 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.294+0000 7f4efea67640 10 monclient: build_initial_monmap 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.294+0000 7f4efea67640 10 monclient: monmap: 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:49:20.295441+0000 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:49:20.295441+0000 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:49:20.302 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:20.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.296+0000 7f4efea67640 1 Processor -- start 2024-03-29T11:49:20.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.297+0000 7f4efea67640 1 -- start start 2024-03-29T11:49:20.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.297+0000 7f4efea67640 10 monclient: init 2024-03-29T11:49:20.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.297+0000 7f4efea67640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:20.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.297+0000 7f4efea67640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:49:20.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.297+0000 7f4efea67640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef0034850 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:20.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.297+0000 7f4efea67640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef00c9550 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:20.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.299+0000 7f4efea67640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef002aac0 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:20.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.299+0000 7f4efea67640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef002c440 con 0x7f4ef00c9550 2024-03-29T11:49:20.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.299+0000 7f4efea67640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef002aed0 con 0x7f4ef002aac0 2024-03-29T11:49:20.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.299+0000 7f4efea67640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef002b190 con 0x7f4ef0034850 2024-03-29T11:49:20.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.299+0000 7f4efea67640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.299+0000 7f4efea67640 10 monclient(hunting): authenticate will time out at 1373.578822s 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.302+0000 7f4efe266640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4ef002aac0 0x7f4ef00cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:48085/0 (socket says 172.21.15.55:48085) 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.303+0000 7f4efe266640 1 -- 172.21.15.55:0/3753544886 learned_addr learned my addr 172.21.15.55:0/3753544886 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.303+0000 7f4efca63640 1 -- 172.21.15.55:0/3753544886 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420729457 0 0) 0x7f4ef002aed0 con 0x7f4ef002aac0 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.303+0000 7f4efca63640 10 monclient(hunting): _negotiate 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.303+0000 7f4efca63640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.303+0000 7f4efca63640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.303+0000 7f4efca63640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.303+0000 7f4efca63640 1 -- 172.21.15.55:0/3753544886 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee4003520 con 0x7f4ef002aac0 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.303+0000 7f4efca63640 1 -- 172.21.15.55:0/3753544886 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2612951794 0 0) 0x7f4ee4003520 con 0x7f4ef002aac0 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.303+0000 7f4efca63640 1 -- 172.21.15.55:0/3753544886 >> v1:172.21.15.55:6790/0 conn(0x7f4ef0034850 legacy=0x7f4ef00c8e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 1 -- 172.21.15.55:0/3753544886 >> v1:172.21.15.55:6789/0 conn(0x7f4ef00c9550 legacy=0x7f4ef00cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:49:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 10 monclient: _finish_hunting 0 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 1 monclient: found mon.noname-c 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 1 -- 172.21.15.55:0/3753544886 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef0033060 con 0x7f4ef002aac0 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 10 monclient: _finish_auth 0 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 10 monclient: _check_auth_tickets 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 1 -- 172.21.15.55:0/3753544886 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4ef804f730 con 0x7f4ef002aac0 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efca63640 10 monclient: dump: 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:20.315 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efea67640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.305+0000 7f4efea67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.307+0000 7f4efca63640 1 -- 172.21.15.55:0/3753544886 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ef80008d0 con 0x7f4ef002aac0 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.307+0000 7f4efca63640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.307+0000 7f4efca63640 1 -- 172.21.15.55:0/3753544886 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4ef80758b0 con 0x7f4ef002aac0 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.307+0000 7f4efca63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.307+0000 7f4efca63640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:20.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.307+0000 7f4efca63640 10 monclient: dump: 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.307+0000 7f4efea67640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:20.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.307+0000 7f4efea67640 4 set_mon_vals no callback set 2024-03-29T11:49:20.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.309+0000 7f4efea67640 10 monclient: shutdown 2024-03-29T11:49:20.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.309+0000 7f4efea67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:20.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.309+0000 7f4efea67640 1 -- 172.21.15.55:0/3753544886 >> v1:172.21.15.171:6789/0 conn(0x7f4ef002aac0 legacy=0x7f4ef00cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:20.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.309+0000 7f4efea67640 1 -- 172.21.15.55:0/3753544886 shutdown_connections 2024-03-29T11:49:20.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.309+0000 7f4efea67640 1 -- 172.21.15.55:0/3753544886 >> 172.21.15.55:0/3753544886 conn(0x7f4ef00058f0 msgr2=0x7f4ef0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:20.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.310+0000 7f4efea67640 1 -- 172.21.15.55:0/3753544886 shutdown_connections 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.310+0000 7f4efea67640 1 -- 172.21.15.55:0/3753544886 wait complete. 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.311+0000 7f4efea67640 10 monclient: build_initial_monmap 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.311+0000 7f4efea67640 10 monclient: monmap: 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.311+0000 7f4efea67640 1 librados: starting msgr at 2024-03-29T11:49:20.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.312+0000 7f4efea67640 1 librados: starting objecter 2024-03-29T11:49:20.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.312+0000 7f4efea67640 1 Processor -- start 2024-03-29T11:49:20.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.312+0000 7f4efea67640 1 -- start start 2024-03-29T11:49:20.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.312+0000 7f4efea67640 1 librados: setting wanted keys 2024-03-29T11:49:20.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.312+0000 7f4efea67640 1 librados: calling monclient init 2024-03-29T11:49:20.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.312+0000 7f4efea67640 10 monclient: init 2024-03-29T11:49:20.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.313+0000 7f4efea67640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:20.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.313+0000 7f4efea67640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:49:20.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.313+0000 7f4efea67640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef002aac0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:20.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.314+0000 7f4efea67640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef0034850 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.314+0000 7f4efea67640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef00c9550 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.314+0000 7f4efea67640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef00c2440 con 0x7f4ef002aac0 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.314+0000 7f4efea67640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef00c2720 con 0x7f4ef0034850 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.314+0000 7f4efea67640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef0038070 con 0x7f4ef00c9550 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.314+0000 7f4efea67640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.314+0000 7f4efea67640 10 monclient(hunting): authenticate will time out at 1373.594447s 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4efe266640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f4ef00c9550 0x7f4ef00c1040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:55563/0 (socket says 172.21.15.55:55563) 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4efe266640 1 -- 172.21.15.55:0/1487633434 learned_addr learned my addr 172.21.15.55:0/1487633434 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4eee7fc640 10 client.?.objecter ms_handle_connect 0x7f4ef00c9550 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4eee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436153307 0 0) 0x7f4ef0038070 con 0x7f4ef00c9550 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4eee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4eee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:20.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4eee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:20.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4eee7fc640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:49:20.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.318+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4edc004eb0 con 0x7f4ef00c9550 2024-03-29T11:49:20.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.320+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2253107393 0 0) 0x7f4edc004eb0 con 0x7f4ef00c9550 2024-03-29T11:49:20.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.320+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ef0038070 con 0x7f4ef00c9550 2024-03-29T11:49:20.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.320+0000 7f4eee7fc640 10 client.?.objecter ms_handle_connect 0x7f4ef002aac0 2024-03-29T11:49:20.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.320+0000 7f4eee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.320+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4ef804f400 con 0x7f4ef00c9550 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.320+0000 7f4eee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.320+0000 7f4eee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.320+0000 7f4eee7fc640 10 monclient(hunting): dump: 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:20.330 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.320+0000 7f4eee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3149430185 0 0) 0x7f4ef0038070 con 0x7f4ef00c9550 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 >> v1:172.21.15.171:6789/0 conn(0x7f4ef0034850 legacy=0x7f4ef00ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 >> v1:172.21.15.55:6789/0 conn(0x7f4ef002aac0 legacy=0x7f4ef00c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 1 monclient: found mon.c 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef00c5620 con 0x7f4ef00c9550 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ef804f730 con 0x7f4ef00c9550 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4efea67640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:49:20.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4efea67640 10 monclient: _renew_subs 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.321+0000 7f4efea67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef016c810 con 0x7f4ef00c9550 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4efea67640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4efea67640 10 monclient: _renew_subs 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4efea67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef016cb80 con 0x7f4ef00c9550 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4efea67640 1 librados: init done 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4eee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4ef8067d90 con 0x7f4ef00c9550 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4eeffff640 4 set_mon_vals no callback set 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.322+0000 7f4eee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.323+0000 7f4eee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.323+0000 7f4eee7fc640 10 monclient: dump: 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:20.332 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:20.333 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:20.333 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:20.333 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:20.333 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:20.333 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:20.333 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:20.333 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:20.333 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:20.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.325+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f4ef8067ef0 con 0x7f4ef00c9550 2024-03-29T11:49:20.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.326+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f4ef80ff5c0 con 0x7f4ef00c9550 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.326+0000 7f4eee7fc640 10 client.4496.objecter ms_dispatch 0x7f4ef8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.326+0000 7f4eee7fc640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.326+0000 7f4efda65640 10 monclient: get_auth_request con 0x7f4edc06dc40 auth_method 0 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.326+0000 7f4eee7fc640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.326+0000 7f4eee7fc640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.327+0000 7f4eee7fc640 10 client.4496.objecter ms_handle_connect 0x7f4edc06dc40 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.328+0000 7f4efea67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.328+0000 7f4efea67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.328+0000 7f4efea67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.328+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee0005080 con 0x7f4ef00c9550 2024-03-29T11:49:20.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.339+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ef80caea0 con 0x7f4ef00c9550 2024-03-29T11:49:20.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.339+0000 7f4eee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:20.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.339+0000 7f4eee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:20.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.672+0000 7f4efea67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:20.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.672+0000 7f4efea67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:20.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.672+0000 7f4efea67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:20.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.672+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ee0005330 con 0x7f4ef00c9550 2024-03-29T11:49:20.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.674+0000 7f4eee7fc640 1 -- 172.21.15.55:0/1487633434 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f4ef80ccdd0 con 0x7f4ef00c9550 2024-03-29T11:49:20.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.674+0000 7f4eee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:20.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.674+0000 7f4eee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:20.680 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:49:20.680 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:20.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 10 librados: watch_flush enter 2024-03-29T11:49:20.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 10 librados: watch_flush exit 2024-03-29T11:49:20.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T11:49:20.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 10 client.4496.objecter successfully canceled tick 2024-03-29T11:49:20.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 >> v1:172.21.15.171:6816/1463197163 conn(0x7f4edc06dc40 legacy=0x7f4edc070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:20.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 10 monclient: shutdown 2024-03-29T11:49:20.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:20.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 >> v1:172.21.15.55:6790/0 conn(0x7f4ef00c9550 legacy=0x7f4ef00c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:20.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 shutdown_connections 2024-03-29T11:49:20.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.678+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 >> 172.21.15.55:0/1487633434 conn(0x7f4ef00058f0 msgr2=0x7f4ef0029610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.681+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 shutdown_connections 2024-03-29T11:49:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.682+0000 7f4efea67640 1 -- 172.21.15.55:0/1487633434 wait complete. 2024-03-29T11:49:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:20.682+0000 7f4efea67640 1 librados: shutdown 2024-03-29T11:49:20.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:27.513 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:27.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.659+0000 7f7a33e4a640 10 monclient: get_monmap_and_config 2024-03-29T11:49:27.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.659+0000 7f7a33e4a640 10 monclient: build_initial_monmap 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.659+0000 7f7a33e4a640 10 monclient: monmap: 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:49:27.659673+0000 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:49:27.659673+0000 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:49:27.665 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:27.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.662+0000 7f7a33e4a640 1 Processor -- start 2024-03-29T11:49:27.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.663+0000 7f7a33e4a640 1 -- start start 2024-03-29T11:49:27.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.663+0000 7f7a33e4a640 10 monclient: init 2024-03-29T11:49:27.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:27.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:49:27.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 10 monclient(hunting): picked mon.noname-c con 0x7f7a2c1422b0 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 10 monclient(hunting): picked mon.noname-a con 0x7f7a2c142690 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 10 monclient(hunting): picked mon.noname-b con 0x7f7a2c0b17d0 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c142c80 con 0x7f7a2c142690 2024-03-29T11:49:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c0aae40 con 0x7f7a2c1422b0 2024-03-29T11:49:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c13fbd0 con 0x7f7a2c0b17d0 2024-03-29T11:49:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.664+0000 7f7a33e4a640 10 monclient(hunting): authenticate will time out at 1380.943051s 2024-03-29T11:49:27.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.668+0000 7f7a31be6640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7a2c1422b0 0x7f7a2c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:45041/0 (socket says 172.21.15.55:45041) 2024-03-29T11:49:27.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.668+0000 7f7a31be6640 1 -- 172.21.15.55:0/331900212 learned_addr learned my addr 172.21.15.55:0/331900212 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:27.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.669+0000 7f7a30be4640 1 -- 172.21.15.55:0/331900212 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800525629 0 0) 0x7f7a2c0aae40 con 0x7f7a2c1422b0 2024-03-29T11:49:27.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.669+0000 7f7a30be4640 10 monclient(hunting): _negotiate 2024-03-29T11:49:27.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.669+0000 7f7a30be4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:27.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.669+0000 7f7a30be4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:27.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.669+0000 7f7a30be4640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:49:27.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.669+0000 7f7a30be4640 1 -- 172.21.15.55:0/331900212 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a18003520 con 0x7f7a2c1422b0 2024-03-29T11:49:27.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.669+0000 7f7a30be4640 1 -- 172.21.15.55:0/331900212 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3285393572 0 0) 0x7f7a18003520 con 0x7f7a2c1422b0 2024-03-29T11:49:27.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.669+0000 7f7a30be4640 1 -- 172.21.15.55:0/331900212 >> v1:172.21.15.55:6790/0 conn(0x7f7a2c0b17d0 legacy=0x7f7a2c0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:49:27.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 1 -- 172.21.15.55:0/331900212 >> v1:172.21.15.55:6789/0 conn(0x7f7a2c142690 legacy=0x7f7a2c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: _finish_hunting 0 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 1 monclient: found mon.noname-c 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 1 -- 172.21.15.55:0/331900212 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c151940 con 0x7f7a2c1422b0 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: _finish_auth 0 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: _check_auth_tickets 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 1 -- 172.21.15.55:0/331900212 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7a28001690 con 0x7f7a2c1422b0 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:27.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: dump: 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 1 -- 172.21.15.55:0/331900212 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a280017f0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:27.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 1 -- 172.21.15.55:0/331900212 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7a28001c40 con 0x7f7a2c1422b0 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.670+0000 7f7a30be4640 10 monclient: dump: 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.671+0000 7f7a33e4a640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.671+0000 7f7a33e4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.671+0000 7f7a33e4a640 4 set_mon_vals no callback set 2024-03-29T11:49:27.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.671+0000 7f7a33e4a640 10 monclient: shutdown 2024-03-29T11:49:27.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.671+0000 7f7a33e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:27.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.671+0000 7f7a33e4a640 1 -- 172.21.15.55:0/331900212 >> v1:172.21.15.171:6789/0 conn(0x7f7a2c1422b0 legacy=0x7f7a2c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:27.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.672+0000 7f7a33e4a640 1 -- 172.21.15.55:0/331900212 shutdown_connections 2024-03-29T11:49:27.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.672+0000 7f7a33e4a640 1 -- 172.21.15.55:0/331900212 >> 172.21.15.55:0/331900212 conn(0x7f7a2c066a00 msgr2=0x7f7a2c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:27.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.672+0000 7f7a33e4a640 1 -- 172.21.15.55:0/331900212 shutdown_connections 2024-03-29T11:49:27.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.672+0000 7f7a33e4a640 1 -- 172.21.15.55:0/331900212 wait complete. 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.673+0000 7f7a33e4a640 10 monclient: build_initial_monmap 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.673+0000 7f7a33e4a640 10 monclient: monmap: 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.674+0000 7f7a33e4a640 1 librados: starting msgr at 2024-03-29T11:49:27.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.674+0000 7f7a33e4a640 1 librados: starting objecter 2024-03-29T11:49:27.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.674+0000 7f7a33e4a640 1 Processor -- start 2024-03-29T11:49:27.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.674+0000 7f7a33e4a640 1 -- start start 2024-03-29T11:49:27.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.674+0000 7f7a33e4a640 1 librados: setting wanted keys 2024-03-29T11:49:27.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.674+0000 7f7a33e4a640 1 librados: calling monclient init 2024-03-29T11:49:27.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.674+0000 7f7a33e4a640 10 monclient: init 2024-03-29T11:49:27.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.675+0000 7f7a33e4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:27.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.675+0000 7f7a33e4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:49:27.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.675+0000 7f7a33e4a640 10 monclient(hunting): picked mon.noname-a con 0x7f7a2c0b17d0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:27.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.680+0000 7f7a33e4a640 10 monclient(hunting): picked mon.noname-c con 0x7f7a2c1422b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:27.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.680+0000 7f7a313e5640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f7a2c1422b0 0x7f7a2c12e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:55331/0 (socket says 172.21.15.55:55331) 2024-03-29T11:49:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.684+0000 7f7a313e5640 1 -- 172.21.15.55:0/1818560499 learned_addr learned my addr 172.21.15.55:0/1818560499 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.684+0000 7f7a33e4a640 10 monclient(hunting): picked mon.noname-b con 0x7f7a2c142690 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.684+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c1df610 con 0x7f7a2c0b17d0 2024-03-29T11:49:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.684+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c12f680 con 0x7f7a2c142690 2024-03-29T11:49:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c06edf0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a33e4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a33e4a640 10 monclient(hunting): authenticate will time out at 1380.963958s 2024-03-29T11:49:27.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 client.?.objecter ms_handle_connect 0x7f7a2c1422b0 2024-03-29T11:49:27.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:27.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 client.?.objecter ms_handle_connect 0x7f7a2c142690 2024-03-29T11:49:27.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:27.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474365377 0 0) 0x7f7a2c12f680 con 0x7f7a2c142690 2024-03-29T11:49:27.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:49:27.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a10004eb0 con 0x7f7a2c142690 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161162113 0 0) 0x7f7a2c06edf0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2c12f680 con 0x7f7a2c1422b0 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 client.?.objecter ms_handle_connect 0x7f7a2c0b17d0 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.685+0000 7f7a227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2688015946 0 0) 0x7f7a2c12f680 con 0x7f7a2c1422b0 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a2c06edf0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7a1c003f20 con 0x7f7a2c1422b0 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 10 monclient(hunting): dump: 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:27.697 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260445905 0 0) 0x7f7a2c1df610 con 0x7f7a2c0b17d0 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.686+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2c12f680 con 0x7f7a2c0b17d0 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3527342705 0 0) 0x7f7a2c06edf0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 >> v1:172.21.15.171:6789/0 conn(0x7f7a2c142690 legacy=0x7f7a2c12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 >> v1:172.21.15.55:6789/0 conn(0x7f7a2c0b17d0 legacy=0x7f7a2c131820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:49:27.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 1 monclient: found mon.c 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c06de90 con 0x7f7a2c1422b0 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 10 monclient: _finish_auth 0 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a33e4a640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a33e4a640 10 monclient: _renew_subs 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a33e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c06e0e0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a33e4a640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:27.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a33e4a640 10 monclient: _renew_subs 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a33e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c1df170 con 0x7f7a2c1422b0 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.687+0000 7f7a33e4a640 1 librados: init done 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.689+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a1c0044b0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.689+0000 7f7a227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.689+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f7a1c001070 con 0x7f7a2c1422b0 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.689+0000 7f7a227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.689+0000 7f7a227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.689+0000 7f7a227fc640 10 monclient: dump: 2024-03-29T11:49:27.700 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:27.701 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:27.701 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:27.701 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:27.701 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:27.701 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:27.701 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:27.701 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:27.701 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:27.701 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:27.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.691+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f7a1c0011d0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.691+0000 7f7a23fff640 4 set_mon_vals no callback set 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.692+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f7a1c0849d0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.692+0000 7f7a227fc640 10 client.4499.objecter ms_dispatch 0x7f7a2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.692+0000 7f7a227fc640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.692+0000 7f7a227fc640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.692+0000 7f7a227fc640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.692+0000 7f7a31be6640 10 monclient: get_auth_request con 0x7f7a1006dd30 auth_method 0 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.694+0000 7f7a33e4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.694+0000 7f7a33e4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.694+0000 7f7a33e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.694+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a14005080 con 0x7f7a2c1422b0 2024-03-29T11:49:27.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.696+0000 7f7a227fc640 10 client.4499.objecter ms_handle_connect 0x7f7a1006dd30 2024-03-29T11:49:27.707 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.702+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a1c0500e0 con 0x7f7a2c1422b0 2024-03-29T11:49:27.707 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.702+0000 7f7a227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:27.707 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:27.702+0000 7f7a227fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:28.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.033+0000 7f7a33e4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:28.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.033+0000 7f7a33e4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:28.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.033+0000 7f7a33e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:28.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.033+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a14005330 con 0x7f7a2c1422b0 2024-03-29T11:49:28.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.034+0000 7f7a227fc640 1 -- 172.21.15.55:0/1818560499 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f7a1c0521e0 con 0x7f7a2c1422b0 2024-03-29T11:49:28.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.034+0000 7f7a227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:28.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.034+0000 7f7a227fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:28.042 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:49:28.042 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:28.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.040+0000 7f7a33e4a640 10 librados: watch_flush enter 2024-03-29T11:49:28.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.040+0000 7f7a33e4a640 10 librados: watch_flush exit 2024-03-29T11:49:28.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.040+0000 7f7a33e4a640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:49:28.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.040+0000 7f7a33e4a640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:49:28.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.041+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 >> v1:172.21.15.171:6816/1463197163 conn(0x7f7a1006dd30 legacy=0x7f7a100701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:28.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.041+0000 7f7a33e4a640 10 monclient: shutdown 2024-03-29T11:49:28.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.041+0000 7f7a33e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:28.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.041+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 >> v1:172.21.15.55:6790/0 conn(0x7f7a2c1422b0 legacy=0x7f7a2c12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:28.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.041+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 shutdown_connections 2024-03-29T11:49:28.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.041+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 >> 172.21.15.55:0/1818560499 conn(0x7f7a2c066a00 msgr2=0x7f7a2c13c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:28.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.041+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 shutdown_connections 2024-03-29T11:49:28.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.042+0000 7f7a33e4a640 1 -- 172.21.15.55:0/1818560499 wait complete. 2024-03-29T11:49:28.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:28.042+0000 7f7a33e4a640 1 librados: shutdown 2024-03-29T11:49:28.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:34.870 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:35.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.019+0000 7f26da172640 10 monclient: get_monmap_and_config 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.019+0000 7f26da172640 10 monclient: build_initial_monmap 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.019+0000 7f26da172640 10 monclient: monmap: 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:49:35.020620+0000 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:49:35.020620+0000 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:49:35.026 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:35.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.024+0000 7f26da172640 1 Processor -- start 2024-03-29T11:49:35.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.024+0000 7f26da172640 1 -- start start 2024-03-29T11:49:35.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.024+0000 7f26da172640 10 monclient: init 2024-03-29T11:49:35.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.025+0000 7f26da172640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:35.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.026+0000 7f26da172640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:49:35.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.026+0000 7f26da172640 10 monclient(hunting): picked mon.noname-a con 0x7f26d41457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:35.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.026+0000 7f26da172640 10 monclient(hunting): picked mon.noname-c con 0x7f26d413f650 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:35.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.027+0000 7f26da172640 10 monclient(hunting): picked mon.noname-b con 0x7f26d40a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:35.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.027+0000 7f26da172640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d40aaef0 con 0x7f26d41457f0 2024-03-29T11:49:35.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.027+0000 7f26da172640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d4145bd0 con 0x7f26d413f650 2024-03-29T11:49:35.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.027+0000 7f26da172640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d4145eb0 con 0x7f26d40a9d70 2024-03-29T11:49:35.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.027+0000 7f26da172640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:35.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.027+0000 7f26da172640 10 monclient(hunting): authenticate will time out at 1388.306723s 2024-03-29T11:49:35.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.030+0000 7f26d37fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f26d41457f0 0x7f26d413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:38681/0 (socket says 172.21.15.55:38681) 2024-03-29T11:49:35.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.030+0000 7f26d37fe640 1 -- 172.21.15.55:0/1336902638 learned_addr learned my addr 172.21.15.55:0/1336902638 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:35.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 1 -- 172.21.15.55:0/1336902638 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503692997 0 0) 0x7f26d4145bd0 con 0x7f26d413f650 2024-03-29T11:49:35.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 1 -- 172.21.15.55:0/1336902638 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26bc003520 con 0x7f26d413f650 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 1 -- 172.21.15.55:0/1336902638 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2793696084 0 0) 0x7f26bc003520 con 0x7f26d413f650 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 1 -- 172.21.15.55:0/1336902638 >> v1:172.21.15.55:6790/0 conn(0x7f26d40a9d70 legacy=0x7f26d414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 1 -- 172.21.15.55:0/1336902638 >> v1:172.21.15.55:6789/0 conn(0x7f26d41457f0 legacy=0x7f26d413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 1 monclient: found mon.noname-c 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 1 -- 172.21.15.55:0/1336902638 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d4146340 con 0x7f26d413f650 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient: _finish_auth 0 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:35.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 1 -- 172.21.15.55:0/1336902638 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f26c4001690 con 0x7f26d413f650 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26d27fc640 10 monclient: dump: 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26da172640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.031+0000 7f26da172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.032+0000 7f26d27fc640 1 -- 172.21.15.55:0/1336902638 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26c4001860 con 0x7f26d413f650 2024-03-29T11:49:35.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.032+0000 7f26d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.032+0000 7f26d27fc640 1 -- 172.21.15.55:0/1336902638 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f26c4001cb0 con 0x7f26d413f650 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.032+0000 7f26d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.032+0000 7f26d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.032+0000 7f26d27fc640 10 monclient: dump: 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.032+0000 7f26da172640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:35.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.032+0000 7f26da172640 4 set_mon_vals no callback set 2024-03-29T11:49:35.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.036+0000 7f26da172640 10 monclient: shutdown 2024-03-29T11:49:35.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.036+0000 7f26da172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:35.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.036+0000 7f26da172640 1 -- 172.21.15.55:0/1336902638 >> v1:172.21.15.171:6789/0 conn(0x7f26d413f650 legacy=0x7f26d40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:35.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.036+0000 7f26da172640 1 -- 172.21.15.55:0/1336902638 shutdown_connections 2024-03-29T11:49:35.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.036+0000 7f26da172640 1 -- 172.21.15.55:0/1336902638 >> 172.21.15.55:0/1336902638 conn(0x7f26d4066a00 msgr2=0x7f26d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:35.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.037+0000 7f26da172640 1 -- 172.21.15.55:0/1336902638 shutdown_connections 2024-03-29T11:49:35.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.037+0000 7f26da172640 1 -- 172.21.15.55:0/1336902638 wait complete. 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.038+0000 7f26da172640 10 monclient: build_initial_monmap 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.039+0000 7f26da172640 10 monclient: monmap: 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:49:35.045 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:35.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.039+0000 7f26da172640 1 librados: starting msgr at 2024-03-29T11:49:35.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.039+0000 7f26da172640 1 librados: starting objecter 2024-03-29T11:49:35.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.040+0000 7f26da172640 1 Processor -- start 2024-03-29T11:49:35.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.040+0000 7f26da172640 1 -- start start 2024-03-29T11:49:35.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.040+0000 7f26da172640 1 librados: setting wanted keys 2024-03-29T11:49:35.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.040+0000 7f26da172640 1 librados: calling monclient init 2024-03-29T11:49:35.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.040+0000 7f26da172640 10 monclient: init 2024-03-29T11:49:35.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.041+0000 7f26da172640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:35.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.041+0000 7f26da172640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:49:35.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.041+0000 7f26da172640 10 monclient(hunting): picked mon.noname-c con 0x7f26d40a9d70 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:35.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.042+0000 7f26d37fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f26d40a9d70 0x7f26d41d30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:45328/0 (socket says 172.21.15.55:45328) 2024-03-29T11:49:35.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.042+0000 7f26d37fe640 1 -- 172.21.15.55:0/873876945 learned_addr learned my addr 172.21.15.55:0/873876945 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:35.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.042+0000 7f26da172640 10 monclient(hunting): picked mon.noname-a con 0x7f26d41457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:35.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.044+0000 7f26bbfff640 10 client.?.objecter ms_handle_connect 0x7f26d40a9d70 2024-03-29T11:49:35.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.044+0000 7f26bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:35.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.044+0000 7f26da172640 10 monclient(hunting): picked mon.noname-b con 0x7f26d4141f50 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:35.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.044+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d41d3980 con 0x7f26d41457f0 2024-03-29T11:49:35.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.044+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d406ea40 con 0x7f26d4141f50 2024-03-29T11:49:35.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.044+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d41d0730 con 0x7f26d40a9d70 2024-03-29T11:49:35.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.044+0000 7f26da172640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:35.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.044+0000 7f26da172640 10 monclient(hunting): authenticate will time out at 1388.324575s 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.045+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748578681 0 0) 0x7f26d41d0730 con 0x7f26d40a9d70 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.045+0000 7f26bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.045+0000 7f26bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.045+0000 7f26bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.045+0000 7f26bbfff640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.045+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c0004eb0 con 0x7f26d40a9d70 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.046+0000 7f26bbfff640 10 client.?.objecter ms_handle_connect 0x7f26d41457f0 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.046+0000 7f26bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.046+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2498545451 0 0) 0x7f26c0004eb0 con 0x7f26d40a9d70 2024-03-29T11:49:35.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.047+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26d41d0730 con 0x7f26d40a9d70 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.047+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044114541 0 0) 0x7f26d41d3980 con 0x7f26d41457f0 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.047+0000 7f26bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.047+0000 7f26bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.047+0000 7f26bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.047+0000 7f26bbfff640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.047+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c0004eb0 con 0x7f26d41457f0 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.047+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f26cc001b50 con 0x7f26d40a9d70 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.047+0000 7f26bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 10 monclient(hunting): dump: 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:35.055 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:35.056 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:35.056 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:35.056 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:35.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:35.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 163438077 0 0) 0x7f26d41d0730 con 0x7f26d40a9d70 2024-03-29T11:49:35.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 >> v1:172.21.15.171:6789/0 conn(0x7f26d4141f50 legacy=0x7f26d4138570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 >> v1:172.21.15.55:6789/0 conn(0x7f26d41457f0 legacy=0x7f26d4137e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 1 monclient: found mon.c 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d41e17c0 con 0x7f26d40a9d70 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:35.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26da172640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26da172640 10 monclient: _renew_subs 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26da172640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.048+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d41de790 con 0x7f26d40a9d70 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.049+0000 7f26da172640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.049+0000 7f26da172640 10 monclient: _renew_subs 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.049+0000 7f26da172640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.049+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d41dec70 con 0x7f26d40a9d70 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.049+0000 7f26da172640 1 librados: init done 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.050+0000 7f26da172640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.050+0000 7f26da172640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.050+0000 7f26da172640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:35.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.050+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d4075f10 con 0x7f26d40a9d70 2024-03-29T11:49:35.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26cc001cb0 con 0x7f26d40a9d70 2024-03-29T11:49:35.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:35.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f26cc004dc0 con 0x7f26d40a9d70 2024-03-29T11:49:35.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:35.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:35.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 10 monclient: dump: 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f26cc01add0 con 0x7f26d40a9d70 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f26cc084e70 con 0x7f26d40a9d70 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 10 client.4502.objecter ms_dispatch 0x7f26d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:35.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.053+0000 7f26bbfff640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T11:49:35.062 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.054+0000 7f26d2ffd640 10 monclient: get_auth_request con 0x7f26c006de40 auth_method 0 2024-03-29T11:49:35.062 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.054+0000 7f26d1ffb640 4 set_mon_vals no callback set 2024-03-29T11:49:35.062 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.054+0000 7f26bbfff640 10 client.4502.objecter ms_handle_connect 0x7f26c006de40 2024-03-29T11:49:35.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.063+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26cc0507d0 con 0x7f26d40a9d70 2024-03-29T11:49:35.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.063+0000 7f26bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:35.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.063+0000 7f26bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:35.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.437+0000 7f26da172640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.437+0000 7f26da172640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.437+0000 7f26da172640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.437+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26d40b3680 con 0x7f26d40a9d70 2024-03-29T11:49:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.438+0000 7f26bbfff640 1 -- 172.21.15.55:0/873876945 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f26cc052700 con 0x7f26d40a9d70 2024-03-29T11:49:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.438+0000 7f26bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.438+0000 7f26bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:35.444 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:49:35.445 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:35.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.441+0000 7f26da172640 10 librados: watch_flush enter 2024-03-29T11:49:35.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.442+0000 7f26da172640 10 librados: watch_flush exit 2024-03-29T11:49:35.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.442+0000 7f26da172640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T11:49:35.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.442+0000 7f26da172640 10 client.4502.objecter successfully canceled tick 2024-03-29T11:49:35.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.443+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 >> v1:172.21.15.171:6816/1463197163 conn(0x7f26c006de40 legacy=0x7f26c0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:35.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.443+0000 7f26da172640 10 monclient: shutdown 2024-03-29T11:49:35.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.443+0000 7f26da172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:35.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.443+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 >> v1:172.21.15.55:6790/0 conn(0x7f26d40a9d70 legacy=0x7f26d41d30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:35.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.443+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 shutdown_connections 2024-03-29T11:49:35.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.443+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 >> 172.21.15.55:0/873876945 conn(0x7f26d4066a00 msgr2=0x7f26d413cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:35.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.443+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 shutdown_connections 2024-03-29T11:49:35.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.443+0000 7f26da172640 1 -- 172.21.15.55:0/873876945 wait complete. 2024-03-29T11:49:35.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:35.443+0000 7f26da172640 1 librados: shutdown 2024-03-29T11:49:35.467 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:42.273 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:42.437 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.431+0000 7fac7ec57640 10 monclient: get_monmap_and_config 2024-03-29T11:49:42.437 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.431+0000 7fac7ec57640 10 monclient: build_initial_monmap 2024-03-29T11:49:42.437 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.431+0000 7fac7ec57640 10 monclient: monmap: 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:49:42.432252+0000 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:49:42.432252+0000 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:49:42.438 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:42.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.432+0000 7fac7ec57640 1 Processor -- start 2024-03-29T11:49:42.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.432+0000 7fac7ec57640 1 -- start start 2024-03-29T11:49:42.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.432+0000 7fac7ec57640 10 monclient: init 2024-03-29T11:49:42.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:42.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:49:42.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 10 monclient(hunting): picked mon.noname-c con 0x7fac781422b0 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:42.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 10 monclient(hunting): picked mon.noname-b con 0x7fac78142690 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:42.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 10 monclient(hunting): picked mon.noname-a con 0x7fac780b17d0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:42.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac78142c80 con 0x7fac780b17d0 2024-03-29T11:49:42.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780aae40 con 0x7fac781422b0 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7813fbd0 con 0x7fac78142690 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7ec57640 10 monclient(hunting): authenticate will time out at 1395.713345s 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7c9f3640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fac781422b0 0x7fac7813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:44166/0 (socket says 172.21.15.55:44166) 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.433+0000 7fac7c9f3640 1 -- 172.21.15.55:0/305477219 learned_addr learned my addr 172.21.15.55:0/305477219 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.434+0000 7fac777fe640 1 -- 172.21.15.55:0/305477219 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923159187 0 0) 0x7fac780aae40 con 0x7fac781422b0 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.434+0000 7fac777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.434+0000 7fac777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.434+0000 7fac777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.434+0000 7fac777fe640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:49:42.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.434+0000 7fac777fe640 1 -- 172.21.15.55:0/305477219 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac68003520 con 0x7fac781422b0 2024-03-29T11:49:42.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.434+0000 7fac777fe640 1 -- 172.21.15.55:0/305477219 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2522749414 0 0) 0x7fac68003520 con 0x7fac781422b0 2024-03-29T11:49:42.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 1 -- 172.21.15.55:0/305477219 >> v1:172.21.15.55:6790/0 conn(0x7fac78142690 legacy=0x7fac780b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:42.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 1 -- 172.21.15.55:0/305477219 >> v1:172.21.15.55:6789/0 conn(0x7fac780b17d0 legacy=0x7fac780b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:42.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:49:42.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 1 monclient: found mon.noname-c 2024-03-29T11:49:42.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:42.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:49:42.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 1 -- 172.21.15.55:0/305477219 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78151940 con 0x7fac781422b0 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 10 monclient: _finish_auth 0 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac7ec57640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac7ec57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 1 -- 172.21.15.55:0/305477219 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fac6c001690 con 0x7fac781422b0 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 10 monclient: dump: 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:42.443 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:42.444 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:42.444 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:42.444 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:42.444 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:42.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac7ec57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:42.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 1 -- 172.21.15.55:0/305477219 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac6c0017f0 con 0x7fac781422b0 2024-03-29T11:49:42.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:42.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac7ec57640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:42.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.435+0000 7fac7ec57640 4 set_mon_vals no callback set 2024-03-29T11:49:42.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.436+0000 7fac7ec57640 10 monclient: shutdown 2024-03-29T11:49:42.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.436+0000 7fac7ec57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:42.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.436+0000 7fac777fe640 1 -- 172.21.15.55:0/305477219 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fac6c001bc0 con 0x7fac781422b0 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.436+0000 7fac7ec57640 1 -- 172.21.15.55:0/305477219 >> v1:172.21.15.171:6789/0 conn(0x7fac781422b0 legacy=0x7fac7813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.436+0000 7fac777fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.436+0000 7fac7ec57640 1 -- 172.21.15.55:0/305477219 shutdown_connections 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.436+0000 7fac7ec57640 1 -- 172.21.15.55:0/305477219 >> 172.21.15.55:0/305477219 conn(0x7fac78066a00 msgr2=0x7fac78066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.436+0000 7fac7ec57640 1 -- 172.21.15.55:0/305477219 shutdown_connections 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.436+0000 7fac7ec57640 1 -- 172.21.15.55:0/305477219 wait complete. 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.437+0000 7fac7ec57640 10 monclient: build_initial_monmap 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.437+0000 7fac7ec57640 10 monclient: monmap: 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:42.448 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.438+0000 7fac7ec57640 1 librados: starting msgr at 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.438+0000 7fac7ec57640 1 librados: starting objecter 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.438+0000 7fac7ec57640 1 Processor -- start 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.438+0000 7fac7ec57640 1 -- start start 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.438+0000 7fac7ec57640 1 librados: setting wanted keys 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.438+0000 7fac7ec57640 1 librados: calling monclient init 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.438+0000 7fac7ec57640 10 monclient: init 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 10 monclient(hunting): picked mon.noname-a con 0x7fac780b17d0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 10 monclient(hunting): picked mon.noname-b con 0x7fac781422b0 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 10 monclient(hunting): picked mon.noname-c con 0x7fac78142690 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:42.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780ac7e0 con 0x7fac780b17d0 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780b1260 con 0x7fac781422b0 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780af700 con 0x7fac78142690 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7ec57640 10 monclient(hunting): authenticate will time out at 1395.718917s 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7c9f3640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fac780b17d0 0x7fac7814dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51658/0 (socket says 172.21.15.55:51658) 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac7c9f3640 1 -- 172.21.15.55:0/3436491143 learned_addr learned my addr 172.21.15.55:0/3436491143 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac757fa640 10 client.?.objecter ms_handle_connect 0x7fac781422b0 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac757fa640 10 client.?.objecter ms_handle_connect 0x7fac780b17d0 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.439+0000 7fac757fa640 10 client.?.objecter ms_handle_connect 0x7fac78142690 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.440+0000 7fac757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.440+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307472696 0 0) 0x7fac780b1260 con 0x7fac781422b0 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.440+0000 7fac757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:49:42.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.440+0000 7fac757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:42.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.440+0000 7fac757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:42.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.440+0000 7fac757fa640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:49:42.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.440+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac5c004eb0 con 0x7fac781422b0 2024-03-29T11:49:42.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160319317 0 0) 0x7fac780af700 con 0x7fac78142690 2024-03-29T11:49:42.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:49:42.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:42.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:42.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:49:42.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac780b1260 con 0x7fac78142690 2024-03-29T11:49:42.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735611690 0 0) 0x7fac780ac7e0 con 0x7fac780b17d0 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac780af700 con 0x7fac780b17d0 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2988633582 0 0) 0x7fac5c004eb0 con 0x7fac781422b0 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.441+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac780ac7e0 con 0x7fac781422b0 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fac64001740 con 0x7fac781422b0 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 10 monclient(hunting): dump: 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:42.454 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3465019495 0 0) 0x7fac780ac7e0 con 0x7fac781422b0 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 >> v1:172.21.15.55:6790/0 conn(0x7fac78142690 legacy=0x7fac781e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 >> v1:172.21.15.55:6789/0 conn(0x7fac780b17d0 legacy=0x7fac7814dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 10 monclient: _finish_hunting 0 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 1 monclient: found mon.b 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:42.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7813c5a0 con 0x7fac781422b0 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 10 monclient: _finish_auth 0 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 10 monclient: _check_auth_tickets 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.442+0000 7fac757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac7ec57640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac7ec57640 10 monclient: _renew_subs 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac7ec57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac780af150 con 0x7fac781422b0 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac7ec57640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac7ec57640 10 monclient: _renew_subs 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac7ec57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7813bd60 con 0x7fac781422b0 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac7ec57640 1 librados: init done 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac640018a0 con 0x7fac781422b0 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fac640037b0 con 0x7fac781422b0 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:42.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac757fa640 10 monclient: dump: 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.443+0000 7fac76ffd640 4 set_mon_vals no callback set 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.444+0000 7fac7ec57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.445+0000 7fac7ec57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.445+0000 7fac7ec57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:42.457 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.445+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac58005080 con 0x7fac781422b0 2024-03-29T11:49:42.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.445+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fac64003910 con 0x7fac781422b0 2024-03-29T11:49:42.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.446+0000 7fac7c9f3640 10 monclient: get_auth_request con 0x7fac5c06dc30 auth_method 0 2024-03-29T11:49:42.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.446+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fac640836d0 con 0x7fac781422b0 2024-03-29T11:49:42.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.446+0000 7fac757fa640 10 client.4486.objecter ms_dispatch 0x7fac78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:42.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.446+0000 7fac757fa640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:42.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.446+0000 7fac757fa640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:42.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.446+0000 7fac757fa640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-29T11:49:42.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.446+0000 7fac757fa640 10 client.4486.objecter ms_handle_connect 0x7fac5c06dc30 2024-03-29T11:49:42.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.453+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac6404efb0 con 0x7fac781422b0 2024-03-29T11:49:42.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.453+0000 7fac757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:42.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.453+0000 7fac757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:42.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.875+0000 7fac7ec57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:42.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.875+0000 7fac7ec57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:42.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.875+0000 7fac7ec57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:42.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.875+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac58005330 con 0x7fac781422b0 2024-03-29T11:49:42.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.876+0000 7fac757fa640 1 -- 172.21.15.55:0/3436491143 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fac64050ee0 con 0x7fac781422b0 2024-03-29T11:49:42.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.876+0000 7fac757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:42.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.876+0000 7fac757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:42.883 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:49:42.883 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:42.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.881+0000 7fac7ec57640 10 librados: watch_flush enter 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.881+0000 7fac7ec57640 10 librados: watch_flush exit 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.881+0000 7fac7ec57640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.881+0000 7fac7ec57640 10 client.4486.objecter successfully canceled tick 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.881+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 >> v1:172.21.15.171:6816/1463197163 conn(0x7fac5c06dc30 legacy=0x7fac5c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.881+0000 7fac7ec57640 10 monclient: shutdown 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.881+0000 7fac7ec57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.881+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 >> v1:172.21.15.171:6789/0 conn(0x7fac781422b0 legacy=0x7fac780afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.882+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 shutdown_connections 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.882+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 >> 172.21.15.55:0/3436491143 conn(0x7fac78066a00 msgr2=0x7fac780b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.882+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 shutdown_connections 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.882+0000 7fac7ec57640 1 -- 172.21.15.55:0/3436491143 wait complete. 2024-03-29T11:49:42.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:42.882+0000 7fac7ec57640 1 librados: shutdown 2024-03-29T11:49:42.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:49.713 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.860+0000 7f92fe64e640 10 monclient: get_monmap_and_config 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.860+0000 7f92fe64e640 10 monclient: build_initial_monmap 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.860+0000 7f92fe64e640 10 monclient: monmap: 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:49:49.861935+0000 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:49:49.861935+0000 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:49.867 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:49.868 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:49:49.868 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:49:49.868 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:49.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.861+0000 7f92fe64e640 1 Processor -- start 2024-03-29T11:49:49.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.861+0000 7f92fe64e640 1 -- start start 2024-03-29T11:49:49.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.861+0000 7f92fe64e640 10 monclient: init 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 10 monclient(hunting): picked mon.noname-b con 0x7f92f80743e0 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 10 monclient(hunting): picked mon.noname-a con 0x7f92f80747c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 10 monclient(hunting): picked mon.noname-c con 0x7f92f813e680 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f806f2e0 con 0x7f92f80747c0 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f8070b40 con 0x7f92f813e680 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f8070e20 con 0x7f92f80743e0 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:49.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.863+0000 7f92fe64e640 10 monclient(hunting): authenticate will time out at 1403.143481s 2024-03-29T11:49:49.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f7fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f92f80743e0 0x7f92f806ebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:55268/0 (socket says 172.21.15.55:55268) 2024-03-29T11:49:49.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f7fff640 1 -- 172.21.15.55:0/1474745569 learned_addr learned my addr 172.21.15.55:0/1474745569 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:49.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650892013 0 0) 0x7f92f806f2e0 con 0x7f92f80747c0 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e0003520 con 0x7f92f80747c0 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804971604 0 0) 0x7f92f8070e20 con 0x7f92f80743e0 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.864+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f806f2e0 con 0x7f92f80743e0 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 296146725 0 0) 0x7f92e0003520 con 0x7f92f80747c0 2024-03-29T11:49:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 >> v1:172.21.15.55:6790/0 conn(0x7f92f80743e0 legacy=0x7f92f806ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 >> v1:172.21.15.171:6789/0 conn(0x7f92f813e680 legacy=0x7f92f813eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f8071100 con 0x7f92f80747c0 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92ec001690 con 0x7f92f80747c0 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92fe64e640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92fe64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92f6ffd640 10 monclient: dump: 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:49.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.865+0000 7f92fe64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:49.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92ec0018b0 con 0x7f92f80747c0 2024-03-29T11:49:49.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:49.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92f6ffd640 1 -- 172.21.15.55:0/1474745569 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92ec001c40 con 0x7f92f80747c0 2024-03-29T11:49:49.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92fe64e640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:49.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92fe64e640 4 set_mon_vals no callback set 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92f6ffd640 10 monclient: dump: 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:49.879 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92fe64e640 10 monclient: shutdown 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92fe64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.866+0000 7f92fe64e640 1 -- 172.21.15.55:0/1474745569 >> v1:172.21.15.55:6789/0 conn(0x7f92f80747c0 legacy=0x7f92f813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.867+0000 7f92fe64e640 1 -- 172.21.15.55:0/1474745569 shutdown_connections 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.867+0000 7f92fe64e640 1 -- 172.21.15.55:0/1474745569 >> 172.21.15.55:0/1474745569 conn(0x7f92f80b4200 msgr2=0x7f92f80b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.867+0000 7f92fe64e640 1 -- 172.21.15.55:0/1474745569 shutdown_connections 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.867+0000 7f92fe64e640 1 -- 172.21.15.55:0/1474745569 wait complete. 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.867+0000 7f92fe64e640 10 monclient: build_initial_monmap 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.868+0000 7f92fe64e640 10 monclient: monmap: 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:49:49.880 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.868+0000 7f92fe64e640 1 librados: starting msgr at 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.868+0000 7f92fe64e640 1 librados: starting objecter 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.868+0000 7f92fe64e640 1 Processor -- start 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.868+0000 7f92fe64e640 1 -- start start 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.868+0000 7f92fe64e640 1 librados: setting wanted keys 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.868+0000 7f92fe64e640 1 librados: calling monclient init 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.868+0000 7f92fe64e640 10 monclient: init 2024-03-29T11:49:49.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fe64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fe64e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fe64e640 10 monclient(hunting): picked mon.noname-b con 0x7f92f80743e0 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fe64e640 10 monclient(hunting): picked mon.noname-a con 0x7f92f80747c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fe64e640 10 monclient(hunting): picked mon.noname-c con 0x7f92f813e680 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fe64e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f80abce0 con 0x7f92f80747c0 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fe64e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f81409e0 con 0x7f92f80743e0 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fe64e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f80abf30 con 0x7f92f813e680 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fcbeb640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f92f813e680 0x7f92f81e2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:55504/0 (socket says 172.21.15.55:55504) 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fcbeb640 1 -- 172.21.15.55:0/138163273 learned_addr learned my addr 172.21.15.55:0/138163273 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.869+0000 7f92fe64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 client.?.objecter ms_handle_connect 0x7f92f813e680 2024-03-29T11:49:49.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92fe64e640 10 monclient(hunting): authenticate will time out at 1403.149700s 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 client.?.objecter ms_handle_connect 0x7f92f80747c0 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138853897 0 0) 0x7f92f80abf30 con 0x7f92f813e680 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e4004eb0 con 0x7f92f813e680 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 client.?.objecter ms_handle_connect 0x7f92f80743e0 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843595549 0 0) 0x7f92f80abce0 con 0x7f92f80747c0 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:49.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.870+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f80abf30 con 0x7f92f80747c0 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960477487 0 0) 0x7f92f81409e0 con 0x7f92f80743e0 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f80abce0 con 0x7f92f80743e0 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3539320678 0 0) 0x7f92f80abf30 con 0x7f92f80747c0 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92f81409e0 con 0x7f92f80747c0 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1419567885 0 0) 0x7f92e4004eb0 con 0x7f92f813e680 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92f80abf30 con 0x7f92f813e680 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92ec0015d0 con 0x7f92f80747c0 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:49.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92e8003ee0 con 0x7f92f813e680 2024-03-29T11:49:49.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 394736816 0 0) 0x7f92f80abce0 con 0x7f92f80743e0 2024-03-29T11:49:49.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92e4004eb0 con 0x7f92f80743e0 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92f0001cb0 con 0x7f92f80743e0 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:49.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3909263657 0 0) 0x7f92e4004eb0 con 0x7f92f80743e0 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 >> v1:172.21.15.55:6790/0 conn(0x7f92f813e680 legacy=0x7f92f81e2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.871+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 >> v1:172.21.15.55:6789/0 conn(0x7f92f80747c0 legacy=0x7f92f80accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 1 monclient: found mon.b 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f80ac210 con 0x7f92f80743e0 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92fe64e640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92fe64e640 10 monclient: _renew_subs 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92fe64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:49.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f80ac4f0 con 0x7f92f80743e0 2024-03-29T11:49:49.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92fe64e640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:49.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92fe64e640 10 monclient: _renew_subs 2024-03-29T11:49:49.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92fe64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:49.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f80a9e90 con 0x7f92f80743e0 2024-03-29T11:49:49.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92fe64e640 1 librados: init done 2024-03-29T11:49:49.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92f0004b60 con 0x7f92f80743e0 2024-03-29T11:49:49.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:49.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92f0004e70 con 0x7f92f80743e0 2024-03-29T11:49:49.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:49.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:49.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f4ff9640 10 monclient: dump: 2024-03-29T11:49:49.890 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:49.890 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:49.890 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.872+0000 7f92f67fc640 4 set_mon_vals no callback set 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.874+0000 7f92fe64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.874+0000 7f92fe64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.874+0000 7f92fe64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.874+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d8005080 con 0x7f92f80743e0 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.875+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f92f001ae40 con 0x7f92f80743e0 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.876+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f92f0084d50 con 0x7f92f80743e0 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.876+0000 7f92f4ff9640 10 client.4489.objecter ms_dispatch 0x7f92f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:49.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.876+0000 7f92f4ff9640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:49.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.876+0000 7f92f4ff9640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:49.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.876+0000 7f92f4ff9640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-29T11:49:49.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.876+0000 7f92f77fe640 10 monclient: get_auth_request con 0x7f92e406e120 auth_method 0 2024-03-29T11:49:49.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.877+0000 7f92f4ff9640 10 client.4489.objecter ms_handle_connect 0x7f92e406e120 2024-03-29T11:49:49.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.885+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92f0050630 con 0x7f92f80743e0 2024-03-29T11:49:49.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.885+0000 7f92f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:49.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:49.885+0000 7f92f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:50.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.275+0000 7f92fe64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:50.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.275+0000 7f92fe64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:50.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.275+0000 7f92fe64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:49:50.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.275+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92d8005330 con 0x7f92f80743e0 2024-03-29T11:49:50.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.277+0000 7f92f4ff9640 1 -- 172.21.15.55:0/138163273 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f92f0052560 con 0x7f92f80743e0 2024-03-29T11:49:50.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.277+0000 7f92f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:50.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.277+0000 7f92f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:50.284 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:49:50.284 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:50.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.280+0000 7f92fe64e640 10 librados: watch_flush enter 2024-03-29T11:49:50.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 10 librados: watch_flush exit 2024-03-29T11:49:50.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-29T11:49:50.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 10 client.4489.objecter successfully canceled tick 2024-03-29T11:49:50.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 >> v1:172.21.15.171:6816/1463197163 conn(0x7f92e406e120 legacy=0x7f92e40705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:50.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 10 monclient: shutdown 2024-03-29T11:49:50.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:50.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 >> v1:172.21.15.171:6789/0 conn(0x7f92f80743e0 legacy=0x7f92f812e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:50.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 shutdown_connections 2024-03-29T11:49:50.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 >> 172.21.15.55:0/138163273 conn(0x7f92f80b4200 msgr2=0x7f92f806f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:50.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 shutdown_connections 2024-03-29T11:49:50.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 1 -- 172.21.15.55:0/138163273 wait complete. 2024-03-29T11:49:50.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:50.281+0000 7f92fe64e640 1 librados: shutdown 2024-03-29T11:49:50.302 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:57.108 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:57.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.231+0000 7f89ef724640 10 monclient: get_monmap_and_config 2024-03-29T11:49:57.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.231+0000 7f89ef724640 10 monclient: build_initial_monmap 2024-03-29T11:49:57.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.231+0000 7f89ef724640 10 monclient: monmap: 2024-03-29T11:49:57.238 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:57.238 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:57.238 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:49:57.232802+0000 2024-03-29T11:49:57.238 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:49:57.232802+0000 2024-03-29T11:49:57.238 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:57.238 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:57.239 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:57.239 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:49:57.239 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:49:57.239 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:57.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.232+0000 7f89ef724640 1 Processor -- start 2024-03-29T11:49:57.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.232+0000 7f89ef724640 1 -- start start 2024-03-29T11:49:57.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.232+0000 7f89ef724640 10 monclient: init 2024-03-29T11:49:57.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:57.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 10 monclient(hunting): picked mon.noname-b con 0x7f89e81422b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 10 monclient(hunting): picked mon.noname-a con 0x7f89e8142690 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 10 monclient(hunting): picked mon.noname-c con 0x7f89e80b17d0 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e8142c80 con 0x7f89e8142690 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80aae40 con 0x7f89e80b17d0 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e813fbd0 con 0x7f89e81422b0 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ef724640 10 monclient(hunting): authenticate will time out at 1410.513160s 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ed4c0640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f89e81422b0 0x7f89e813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:56442/0 (socket says 172.21.15.55:56442) 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.233+0000 7f89ed4c0640 1 -- 172.21.15.55:0/1484297883 learned_addr learned my addr 172.21.15.55:0/1484297883 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996344967 0 0) 0x7f89e8142c80 con 0x7f89e8142690 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:57.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d4003520 con 0x7f89e8142690 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351323134 0 0) 0x7f89e813fbd0 con 0x7f89e81422b0 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e8142c80 con 0x7f89e81422b0 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4227566152 0 0) 0x7f89d4003520 con 0x7f89e8142690 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 >> v1:172.21.15.55:6790/0 conn(0x7f89e81422b0 legacy=0x7f89e813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:57.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.234+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 >> v1:172.21.15.171:6789/0 conn(0x7f89e80b17d0 legacy=0x7f89e80b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:49:57.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:49:57.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 1 monclient: found mon.noname-a 2024-03-29T11:49:57.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:57.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:49:57.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e8151940 con 0x7f89e8142690 2024-03-29T11:49:57.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 10 monclient: _finish_auth 0 2024-03-29T11:49:57.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f89d8001690 con 0x7f89e8142690 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 10 monclient: dump: 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:57.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89ef724640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:49:57.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89ef724640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89d80017f0 con 0x7f89e8142690 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89dffff640 1 -- 172.21.15.55:0/1484297883 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f89d8001b80 con 0x7f89e8142690 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89ef724640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.236+0000 7f89ef724640 4 set_mon_vals no callback set 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.237+0000 7f89ef724640 10 monclient: shutdown 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.237+0000 7f89dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.237+0000 7f89dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.237+0000 7f89dffff640 10 monclient: dump: 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:57.254 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.237+0000 7f89ef724640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.237+0000 7f89ef724640 1 -- 172.21.15.55:0/1484297883 >> v1:172.21.15.55:6789/0 conn(0x7f89e8142690 legacy=0x7f89e80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.238+0000 7f89ef724640 1 -- 172.21.15.55:0/1484297883 shutdown_connections 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.238+0000 7f89ef724640 1 -- 172.21.15.55:0/1484297883 >> 172.21.15.55:0/1484297883 conn(0x7f89e8066a00 msgr2=0x7f89e8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.238+0000 7f89ef724640 1 -- 172.21.15.55:0/1484297883 shutdown_connections 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.238+0000 7f89ef724640 1 -- 172.21.15.55:0/1484297883 wait complete. 2024-03-29T11:49:57.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.239+0000 7f89ef724640 10 monclient: build_initial_monmap 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.239+0000 7f89ef724640 10 monclient: monmap: 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.239+0000 7f89ef724640 1 librados: starting msgr at 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.239+0000 7f89ef724640 1 librados: starting objecter 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.239+0000 7f89ef724640 1 Processor -- start 2024-03-29T11:49:57.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.240+0000 7f89ef724640 1 -- start start 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.240+0000 7f89ef724640 1 librados: setting wanted keys 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.240+0000 7f89ef724640 1 librados: calling monclient init 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.240+0000 7f89ef724640 10 monclient: init 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ef724640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ef724640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ef724640 10 monclient(hunting): picked mon.noname-c con 0x7f89e80b17d0 addr v1:172.21.15.55:6790/0 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ed4c0640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f89e80b17d0 0x7f89e812fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:56558/0 (socket says 172.21.15.55:56558) 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ed4c0640 1 -- 172.21.15.55:0/4065767468 learned_addr learned my addr 172.21.15.55:0/4065767468 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ddffb640 10 client.?.objecter ms_handle_connect 0x7f89e80b17d0 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ef724640 10 monclient(hunting): picked mon.noname-b con 0x7f89e81422b0 addr v1:172.21.15.171:6789/0 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ef724640 10 monclient(hunting): picked mon.noname-a con 0x7f89e8142690 addr v1:172.21.15.55:6789/0 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.241+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e812f680 con 0x7f89e8142690 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80abe40 con 0x7f89e81422b0 2024-03-29T11:49:57.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80acfd0 con 0x7f89e80b17d0 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ef724640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ef724640 10 monclient(hunting): authenticate will time out at 1410.521942s 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623815703 0 0) 0x7f89e80acfd0 con 0x7f89e80b17d0 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ddffb640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89cc004eb0 con 0x7f89e80b17d0 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.242+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281460964 0 0) 0x7f89cc004eb0 con 0x7f89e80b17d0 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89e80acfd0 con 0x7f89e80b17d0 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f89e4001840 con 0x7f89e80b17d0 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:57.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 10 monclient(hunting): dump: 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 10 client.?.objecter ms_handle_connect 0x7f89e8142690 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:57.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1311471593 0 0) 0x7f89e80acfd0 con 0x7f89e80b17d0 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 >> v1:172.21.15.171:6789/0 conn(0x7f89e81422b0 legacy=0x7f89e812e860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 >> v1:172.21.15.55:6789/0 conn(0x7f89e8142690 legacy=0x7f89e812ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.243+0000 7f89ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ddffb640 1 monclient: found mon.c 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e80ad2b0 con 0x7f89e80b17d0 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ef724640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ef724640 10 monclient: _renew_subs 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ef724640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:57.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e80ab540 con 0x7f89e80b17d0 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ef724640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.244+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89e4001a10 con 0x7f89e80b17d0 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f89e4004a00 con 0x7f89e80b17d0 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89df7fe640 4 set_mon_vals no callback set 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89ddffb640 10 monclient: dump: 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:49:57.261 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89ef724640 10 monclient: _renew_subs 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89ef724640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e80ac250 con 0x7f89e80b17d0 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.245+0000 7f89ef724640 1 librados: init done 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.246+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f89e4004b60 con 0x7f89e80b17d0 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.246+0000 7f89ef724640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.246+0000 7f89ef724640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.246+0000 7f89ef724640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.246+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e80abc90 con 0x7f89e80b17d0 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.247+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f89e4084a90 con 0x7f89e80b17d0 2024-03-29T11:49:57.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.247+0000 7f89ddffb640 10 client.4520.objecter ms_dispatch 0x7f89e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:57.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.247+0000 7f89ddffb640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:57.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.247+0000 7f89ddffb640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:57.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.247+0000 7f89ddffb640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:49:57.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.248+0000 7f89eccbf640 10 monclient: get_auth_request con 0x7f89cc06db80 auth_method 0 2024-03-29T11:49:57.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.249+0000 7f89ddffb640 10 client.4520.objecter ms_handle_connect 0x7f89cc06db80 2024-03-29T11:49:57.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.253+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89e4050370 con 0x7f89e80b17d0 2024-03-29T11:49:57.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.253+0000 7f89ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:57.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.253+0000 7f89ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:57.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.552+0000 7f89ef724640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:57.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.552+0000 7f89ef724640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:57.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.552+0000 7f89ef724640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:49:57.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.552+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89e80b2270 con 0x7f89e80b17d0 2024-03-29T11:49:57.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.553+0000 7f89ddffb640 1 -- 172.21.15.55:0/4065767468 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f89e40522a0 con 0x7f89e80b17d0 2024-03-29T11:49:57.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.553+0000 7f89ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:57.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.553+0000 7f89ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:57.560 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:49:57.560 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:57.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.555+0000 7f89ef724640 10 librados: watch_flush enter 2024-03-29T11:49:57.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.555+0000 7f89ef724640 10 librados: watch_flush exit 2024-03-29T11:49:57.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.555+0000 7f89ef724640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:49:57.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.555+0000 7f89ef724640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:49:57.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.556+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 >> v1:172.21.15.171:6816/1463197163 conn(0x7f89cc06db80 legacy=0x7f89cc070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:57.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.556+0000 7f89ef724640 10 monclient: shutdown 2024-03-29T11:49:57.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.556+0000 7f89ef724640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:57.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.556+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 >> v1:172.21.15.55:6790/0 conn(0x7f89e80b17d0 legacy=0x7f89e812fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:57.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.556+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 shutdown_connections 2024-03-29T11:49:57.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.556+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 >> 172.21.15.55:0/4065767468 conn(0x7f89e8066a00 msgr2=0x7f89e80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:57.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.557+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 shutdown_connections 2024-03-29T11:49:57.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.557+0000 7f89ef724640 1 -- 172.21.15.55:0/4065767468 wait complete. 2024-03-29T11:49:57.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:49:57.557+0000 7f89ef724640 1 librados: shutdown 2024-03-29T11:49:57.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:04.383 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:04.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.530+0000 7fa32658c640 10 monclient: get_monmap_and_config 2024-03-29T11:50:04.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.530+0000 7fa32658c640 10 monclient: build_initial_monmap 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.530+0000 7fa32658c640 10 monclient: monmap: 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:50:04.531318+0000 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:50:04.531318+0000 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:50:04.537 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:50:04.538 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:04.538 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.531+0000 7fa32658c640 1 Processor -- start 2024-03-29T11:50:04.538 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.531+0000 7fa32658c640 1 -- start start 2024-03-29T11:50:04.538 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.531+0000 7fa32658c640 10 monclient: init 2024-03-29T11:50:04.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 10 monclient(hunting): picked mon.noname-c con 0x7fa3201457f0 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 10 monclient(hunting): picked mon.noname-b con 0x7fa32013f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 10 monclient(hunting): picked mon.noname-a con 0x7fa3200a9d70 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3200aaef0 con 0x7fa3200a9d70 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa320145bd0 con 0x7fa3201457f0 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa320145eb0 con 0x7fa32013f650 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.532+0000 7fa32658c640 10 monclient(hunting): authenticate will time out at 1417.812232s 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.533+0000 7fa317fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fa32013f650 0x7fa3200b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:35572/0 (socket says 172.21.15.55:35572) 2024-03-29T11:50:04.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.533+0000 7fa317fff640 1 -- 172.21.15.55:0/658895642 learned_addr learned my addr 172.21.15.55:0/658895642 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:04.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.536+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353443911 0 0) 0x7fa320145eb0 con 0x7fa32013f650 2024-03-29T11:50:04.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:50:04.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:04.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:04.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:50:04.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa30c003520 con 0x7fa32013f650 2024-03-29T11:50:04.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154746973 0 0) 0x7fa3200aaef0 con 0x7fa3200a9d70 2024-03-29T11:50:04.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:50:04.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:04.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:04.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:50:04.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa320145eb0 con 0x7fa3200a9d70 2024-03-29T11:50:04.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394568345 0 0) 0x7fa30c003520 con 0x7fa32013f650 2024-03-29T11:50:04.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 >> v1:172.21.15.171:6789/0 conn(0x7fa3201457f0 legacy=0x7fa32013ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 >> v1:172.21.15.55:6789/0 conn(0x7fa3200a9d70 legacy=0x7fa32014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa320146340 con 0x7fa32013f650 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa308001690 con 0x7fa32013f650 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:04.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.537+0000 7fa31f7fe640 10 monclient: dump: 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:04.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.538+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3080018b0 con 0x7fa32013f650 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.538+0000 7fa31f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.538+0000 7fa31f7fe640 1 -- 172.21.15.55:0/658895642 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa308001c40 con 0x7fa32013f650 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.538+0000 7fa31f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.538+0000 7fa31f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.538+0000 7fa31f7fe640 10 monclient: dump: 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:04.548 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.539+0000 7fa32658c640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.539+0000 7fa32658c640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.539+0000 7fa32658c640 4 set_mon_vals no callback set 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.541+0000 7fa32658c640 10 monclient: shutdown 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.541+0000 7fa32658c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.541+0000 7fa32658c640 1 -- 172.21.15.55:0/658895642 >> v1:172.21.15.55:6790/0 conn(0x7fa32013f650 legacy=0x7fa3200b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.541+0000 7fa32658c640 1 -- 172.21.15.55:0/658895642 shutdown_connections 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.541+0000 7fa32658c640 1 -- 172.21.15.55:0/658895642 >> 172.21.15.55:0/658895642 conn(0x7fa320066a00 msgr2=0x7fa320066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.541+0000 7fa32658c640 1 -- 172.21.15.55:0/658895642 shutdown_connections 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.542+0000 7fa32658c640 1 -- 172.21.15.55:0/658895642 wait complete. 2024-03-29T11:50:04.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.542+0000 7fa32658c640 10 monclient: build_initial_monmap 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.542+0000 7fa32658c640 10 monclient: monmap: 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.543+0000 7fa32658c640 1 librados: starting msgr at 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.543+0000 7fa32658c640 1 librados: starting objecter 2024-03-29T11:50:04.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.543+0000 7fa32658c640 1 Processor -- start 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.543+0000 7fa32658c640 1 -- start start 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.543+0000 7fa32658c640 1 librados: setting wanted keys 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.543+0000 7fa32658c640 1 librados: calling monclient init 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.543+0000 7fa32658c640 10 monclient: init 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 10 monclient(hunting): picked mon.noname-b con 0x7fa3200a9d70 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 10 monclient(hunting): picked mon.noname-c con 0x7fa32013f650 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 10 monclient(hunting): picked mon.noname-a con 0x7fa3201457f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa320141370 con 0x7fa3201457f0 2024-03-29T11:50:04.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa320144830 con 0x7fa3200a9d70 2024-03-29T11:50:04.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32013c7f0 con 0x7fa32013f650 2024-03-29T11:50:04.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:04.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.544+0000 7fa32658c640 10 monclient(hunting): authenticate will time out at 1417.824161s 2024-03-29T11:50:04.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa317fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fa32013f650 0x7fa320141a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:36176/0 (socket says 172.21.15.55:36176) 2024-03-29T11:50:04.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa317fff640 1 -- 172.21.15.55:0/141444814 learned_addr learned my addr 172.21.15.55:0/141444814 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:04.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 client.?.objecter ms_handle_connect 0x7fa3201457f0 2024-03-29T11:50:04.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:04.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 client.?.objecter ms_handle_connect 0x7fa3200a9d70 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 client.?.objecter ms_handle_connect 0x7fa32013f650 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634469451 0 0) 0x7fa320141370 con 0x7fa3201457f0 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.548+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa300004eb0 con 0x7fa3201457f0 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432966272 0 0) 0x7fa320144830 con 0x7fa3200a9d70 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:50:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3000051d0 con 0x7fa3200a9d70 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215287551 0 0) 0x7fa32013c7f0 con 0x7fa32013f650 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa320144830 con 0x7fa32013f650 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4241080083 0 0) 0x7fa300004eb0 con 0x7fa3201457f0 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa32013c7f0 con 0x7fa3201457f0 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa318001c30 con 0x7fa3201457f0 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): dump: 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:04.560 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4290687131 0 0) 0x7fa320144830 con 0x7fa32013f650 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa300004eb0 con 0x7fa32013f650 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa308001610 con 0x7fa32013f650 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient(hunting): dump: 2024-03-29T11:50:04.561 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3511454101 0 0) 0x7fa32013c7f0 con 0x7fa3201457f0 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 >> v1:172.21.15.55:6790/0 conn(0x7fa32013f650 legacy=0x7fa320141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 >> v1:172.21.15.171:6789/0 conn(0x7fa3200a9d70 legacy=0x7fa320144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:50:04.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 monclient: found mon.a 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32013ddf0 con 0x7fa3201457f0 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa31d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa32658c640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa32658c640 10 monclient: _renew_subs 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa32658c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32013e040 con 0x7fa3201457f0 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa32658c640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa32658c640 10 monclient: _renew_subs 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa32658c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa320140440 con 0x7fa3201457f0 2024-03-29T11:50:04.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.549+0000 7fa32658c640 1 librados: init done 2024-03-29T11:50:04.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.551+0000 7fa32658c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:04.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.551+0000 7fa32658c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:04.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.551+0000 7fa32658c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:04.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.551+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3201449c0 con 0x7fa3201457f0 2024-03-29T11:50:04.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.552+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa318003970 con 0x7fa3201457f0 2024-03-29T11:50:04.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.552+0000 7fa31d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.552+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa318003c80 con 0x7fa3201457f0 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.552+0000 7fa31d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.552+0000 7fa31d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.552+0000 7fa31d7fa640 10 monclient: dump: 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:04.566 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.552+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fa318019c50 con 0x7fa3201457f0 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.553+0000 7fa31effd640 4 set_mon_vals no callback set 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.553+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fa318083bf0 con 0x7fa3201457f0 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.553+0000 7fa31d7fa640 10 client.4515.objecter ms_dispatch 0x7fa320000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.553+0000 7fa31d7fa640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.553+0000 7fa31d7fa640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.553+0000 7fa31ffff640 10 monclient: get_auth_request con 0x7fa30006dfb0 auth_method 0 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.553+0000 7fa31d7fa640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.554+0000 7fa31d7fa640 10 client.4515.objecter ms_handle_connect 0x7fa30006dfb0 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.561+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa31804f550 con 0x7fa3201457f0 2024-03-29T11:50:04.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.561+0000 7fa31d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:04.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.561+0000 7fa31d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:04.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.891+0000 7fa32658c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:04.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.891+0000 7fa32658c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:04.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.891+0000 7fa32658c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:04.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.891+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa320075f10 con 0x7fa3201457f0 2024-03-29T11:50:04.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.893+0000 7fa31d7fa640 1 -- 172.21.15.55:0/141444814 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fa31804f550 con 0x7fa3201457f0 2024-03-29T11:50:04.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.894+0000 7fa31d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:04.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.894+0000 7fa31d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:04.900 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:50:04.900 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.897+0000 7fa32658c640 10 librados: watch_flush enter 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.897+0000 7fa32658c640 10 librados: watch_flush exit 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.897+0000 7fa32658c640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.897+0000 7fa32658c640 10 client.4515.objecter successfully canceled tick 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.898+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 >> v1:172.21.15.171:6816/1463197163 conn(0x7fa30006dfb0 legacy=0x7fa300070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.898+0000 7fa32658c640 10 monclient: shutdown 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.898+0000 7fa32658c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.898+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 >> v1:172.21.15.55:6789/0 conn(0x7fa3201457f0 legacy=0x7fa3201e6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.898+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 shutdown_connections 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.898+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 >> 172.21.15.55:0/141444814 conn(0x7fa320066a00 msgr2=0x7fa3200b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.898+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 shutdown_connections 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.898+0000 7fa32658c640 1 -- 172.21.15.55:0/141444814 wait complete. 2024-03-29T11:50:04.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:04.898+0000 7fa32658c640 1 librados: shutdown 2024-03-29T11:50:04.923 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:11.729 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:11.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.878+0000 7f9cb8443640 10 monclient: get_monmap_and_config 2024-03-29T11:50:11.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.878+0000 7f9cb8443640 10 monclient: build_initial_monmap 2024-03-29T11:50:11.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.878+0000 7f9cb8443640 10 monclient: monmap: 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:50:11.879264+0000 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:50:11.879264+0000 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:50:11.885 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:11.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.881+0000 7f9cb8443640 1 Processor -- start 2024-03-29T11:50:11.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.881+0000 7f9cb8443640 1 -- start start 2024-03-29T11:50:11.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.881+0000 7f9cb8443640 10 monclient: init 2024-03-29T11:50:11.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:11.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:50:11.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 10 monclient(hunting): picked mon.noname-c con 0x7f9cb00b1f60 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:11.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 10 monclient(hunting): picked mon.noname-a con 0x7f9cb0071eb0 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:11.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 10 monclient(hunting): picked mon.noname-b con 0x7f9cb0143d10 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:11.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb006ef80 con 0x7f9cb0071eb0 2024-03-29T11:50:11.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb006f260 con 0x7f9cb00b1f60 2024-03-29T11:50:11.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb00b40c0 con 0x7f9cb0143d10 2024-03-29T11:50:11.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:11.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.883+0000 7f9cb8443640 10 monclient(hunting): authenticate will time out at 1425.162965s 2024-03-29T11:50:11.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.884+0000 7f9cb69e0640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f9cb0143d10 0x7f9cb0146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:51924/0 (socket says 172.21.15.55:51924) 2024-03-29T11:50:11.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.884+0000 7f9cb69e0640 1 -- 172.21.15.55:0/1358870002 learned_addr learned my addr 172.21.15.55:0/1358870002 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:11.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.886+0000 7f9cb51dd640 1 -- 172.21.15.55:0/1358870002 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990737075 0 0) 0x7f9cb00b40c0 con 0x7f9cb0143d10 2024-03-29T11:50:11.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.886+0000 7f9cb51dd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:11.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.886+0000 7f9cb51dd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:11.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.886+0000 7f9cb51dd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:11.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.886+0000 7f9cb51dd640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:50:11.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.886+0000 7f9cb51dd640 1 -- 172.21.15.55:0/1358870002 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c9c003520 con 0x7f9cb0143d10 2024-03-29T11:50:11.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 1 -- 172.21.15.55:0/1358870002 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2412745204 0 0) 0x7f9c9c003520 con 0x7f9cb0143d10 2024-03-29T11:50:11.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 1 -- 172.21.15.55:0/1358870002 >> v1:172.21.15.171:6789/0 conn(0x7f9cb00b1f60 legacy=0x7f9cb0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 1 -- 172.21.15.55:0/1358870002 >> v1:172.21.15.55:6789/0 conn(0x7f9cb0071eb0 legacy=0x7f9cb0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 10 monclient: _finish_hunting 0 2024-03-29T11:50:11.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 1 monclient: found mon.noname-b 2024-03-29T11:50:11.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:11.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:50:11.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 1 -- 172.21.15.55:0/1358870002 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb0155d70 con 0x7f9cb0143d10 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 10 monclient: _finish_auth 0 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 10 monclient: _check_auth_tickets 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb8443640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb8443640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 1 -- 172.21.15.55:0/1358870002 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9ca8001a10 con 0x7f9cb0143d10 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb51dd640 10 monclient: dump: 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:11.897 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.887+0000 7f9cb8443640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.889+0000 7f9cb51dd640 1 -- 172.21.15.55:0/1358870002 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ca8001b70 con 0x7f9cb0143d10 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.889+0000 7f9cb51dd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.889+0000 7f9cb8443640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.889+0000 7f9cb8443640 4 set_mon_vals no callback set 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb51dd640 1 -- 172.21.15.55:0/1358870002 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9ca8001f80 con 0x7f9cb0143d10 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb51dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb51dd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb51dd640 10 monclient: dump: 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:11.898 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:11.899 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:11.899 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:11.899 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:11.899 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:11.899 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:11.899 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:11.899 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:11.899 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:11.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb8443640 10 monclient: shutdown 2024-03-29T11:50:11.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb8443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:11.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb8443640 1 -- 172.21.15.55:0/1358870002 >> v1:172.21.15.55:6790/0 conn(0x7f9cb0143d10 legacy=0x7f9cb0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb8443640 1 -- 172.21.15.55:0/1358870002 shutdown_connections 2024-03-29T11:50:11.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb8443640 1 -- 172.21.15.55:0/1358870002 >> 172.21.15.55:0/1358870002 conn(0x7f9cb0066a00 msgr2=0x7f9cb0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb8443640 1 -- 172.21.15.55:0/1358870002 shutdown_connections 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.890+0000 7f9cb8443640 1 -- 172.21.15.55:0/1358870002 wait complete. 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.891+0000 7f9cb8443640 10 monclient: build_initial_monmap 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.891+0000 7f9cb8443640 10 monclient: monmap: 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:11.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.892+0000 7f9cb8443640 1 librados: starting msgr at 2024-03-29T11:50:11.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.892+0000 7f9cb8443640 1 librados: starting objecter 2024-03-29T11:50:11.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.892+0000 7f9cb8443640 1 Processor -- start 2024-03-29T11:50:11.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.894+0000 7f9cb8443640 1 -- start start 2024-03-29T11:50:11.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.894+0000 7f9cb8443640 1 librados: setting wanted keys 2024-03-29T11:50:11.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.894+0000 7f9cb8443640 1 librados: calling monclient init 2024-03-29T11:50:11.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.894+0000 7f9cb8443640 10 monclient: init 2024-03-29T11:50:11.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:11.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 10 monclient(hunting): picked mon.noname-a con 0x7f9cb00b1f60 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 10 monclient(hunting): picked mon.noname-c con 0x7f9cb0143d10 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 10 monclient(hunting): picked mon.noname-b con 0x7f9cb00ab2a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb00aae10 con 0x7f9cb00b1f60 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb00b1370 con 0x7f9cb00ab2a0 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb0152840 con 0x7f9cb0143d10 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.895+0000 7f9cb8443640 10 monclient(hunting): authenticate will time out at 1425.175003s 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.896+0000 7f9cb59de640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f9cb0143d10 0x7f9cb01e3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:52970/0 (socket says 172.21.15.55:52970) 2024-03-29T11:50:11.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.896+0000 7f9cb59de640 1 -- 172.21.15.55:0/2370609221 learned_addr learned my addr 172.21.15.55:0/2370609221 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.897+0000 7f9ca6ffd640 10 client.?.objecter ms_handle_connect 0x7f9cb0143d10 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.897+0000 7f9ca6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.897+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926829118 0 0) 0x7f9cb0152840 con 0x7f9cb0143d10 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.897+0000 7f9ca6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.897+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.897+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.897+0000 7f9ca6ffd640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.897+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c94004eb0 con 0x7f9cb0143d10 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1578300286 0 0) 0x7f9c94004eb0 con 0x7f9cb0143d10 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cb0152840 con 0x7f9cb0143d10 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9cac001a30 con 0x7f9cb0143d10 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:11.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 monclient(hunting): dump: 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 client.?.objecter ms_handle_connect 0x7f9cb00ab2a0 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648997835 0 0) 0x7f9cb00b1370 con 0x7f9cb00ab2a0 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:11.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.898+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c94004eb0 con 0x7f9cb00ab2a0 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.899+0000 7f9ca6ffd640 10 client.?.objecter ms_handle_connect 0x7f9cb00b1f60 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.899+0000 7f9ca6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.899+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3489311236 0 0) 0x7f9cb0152840 con 0x7f9cb0143d10 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.899+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 >> v1:172.21.15.171:6789/0 conn(0x7f9cb00ab2a0 legacy=0x7f9cb01ee8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.899+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 >> v1:172.21.15.55:6789/0 conn(0x7f9cb00b1f60 legacy=0x7f9cb00aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.899+0000 7f9ca6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.899+0000 7f9ca6ffd640 1 monclient: found mon.c 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9ca6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9ca6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:11.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb01529d0 con 0x7f9cb0143d10 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9ca6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9ca6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9ca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9cb8443640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9cb8443640 10 monclient: _renew_subs 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9cb8443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb00aa9a0 con 0x7f9cb0143d10 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9cb8443640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9cb8443640 10 monclient: _renew_subs 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9cb8443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb01e4770 con 0x7f9cb0143d10 2024-03-29T11:50:11.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.900+0000 7f9cb8443640 1 librados: init done 2024-03-29T11:50:11.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.902+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cac001b90 con 0x7f9cb0143d10 2024-03-29T11:50:11.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.902+0000 7f9ca6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:11.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.902+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9cac0038c0 con 0x7f9cb0143d10 2024-03-29T11:50:11.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.902+0000 7f9ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:11.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.902+0000 7f9ca6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:11.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.902+0000 7f9ca6ffd640 10 monclient: dump: 2024-03-29T11:50:11.913 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:11.913 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:11.913 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.903+0000 7f9cb49dc640 4 set_mon_vals no callback set 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.904+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f9cac003a20 con 0x7f9cb0143d10 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.904+0000 7f9cb8443640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.905+0000 7f9cb8443640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.905+0000 7f9cb8443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.905+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c98005080 con 0x7f9cb0143d10 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.905+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f9cac084a80 con 0x7f9cb0143d10 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.905+0000 7f9ca6ffd640 10 client.4532.objecter ms_dispatch 0x7f9cb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:11.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.905+0000 7f9ca6ffd640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:11.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.905+0000 7f9ca6ffd640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:11.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.905+0000 7f9cb61df640 10 monclient: get_auth_request con 0x7f9c9406dab0 auth_method 0 2024-03-29T11:50:11.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.905+0000 7f9ca6ffd640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:50:11.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.906+0000 7f9ca6ffd640 10 client.4532.objecter ms_handle_connect 0x7f9c9406dab0 2024-03-29T11:50:11.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.913+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9cac050360 con 0x7f9cb0143d10 2024-03-29T11:50:11.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.913+0000 7f9ca6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:11.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:11.913+0000 7f9ca6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:12.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.237+0000 7f9cb8443640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:12.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.237+0000 7f9cb8443640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:12.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.237+0000 7f9cb8443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:12.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.237+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c98005330 con 0x7f9cb0143d10 2024-03-29T11:50:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.239+0000 7f9ca6ffd640 1 -- 172.21.15.55:0/2370609221 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f9cac052290 con 0x7f9cb0143d10 2024-03-29T11:50:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.239+0000 7f9ca6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.239+0000 7f9ca6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:12.245 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:50:12.246 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:12.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.242+0000 7f9cb8443640 10 librados: watch_flush enter 2024-03-29T11:50:12.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.242+0000 7f9cb8443640 10 librados: watch_flush exit 2024-03-29T11:50:12.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.242+0000 7f9cb8443640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:50:12.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.242+0000 7f9cb8443640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:50:12.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.242+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 >> v1:172.21.15.171:6816/1463197163 conn(0x7f9c9406dab0 legacy=0x7f9c9406ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:12.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.242+0000 7f9cb8443640 10 monclient: shutdown 2024-03-29T11:50:12.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.242+0000 7f9cb8443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:12.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.242+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 >> v1:172.21.15.55:6790/0 conn(0x7f9cb0143d10 legacy=0x7f9cb01e3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:12.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.243+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 shutdown_connections 2024-03-29T11:50:12.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.243+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 >> 172.21.15.55:0/2370609221 conn(0x7f9cb0066a00 msgr2=0x7f9cb0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:12.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.244+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 shutdown_connections 2024-03-29T11:50:12.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.244+0000 7f9cb8443640 1 -- 172.21.15.55:0/2370609221 wait complete. 2024-03-29T11:50:12.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:12.244+0000 7f9cb8443640 1 librados: shutdown 2024-03-29T11:50:12.266 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:19.072 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.214+0000 7f2b21424640 10 monclient: get_monmap_and_config 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.214+0000 7f2b21424640 10 monclient: build_initial_monmap 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.214+0000 7f2b21424640 10 monclient: monmap: 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:50:19.214677+0000 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:50:19.214677+0000 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:19.220 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:50:19.221 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:50:19.221 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:19.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.215+0000 7f2b21424640 1 Processor -- start 2024-03-29T11:50:19.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.216+0000 7f2b21424640 1 -- start start 2024-03-29T11:50:19.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.216+0000 7f2b21424640 10 monclient: init 2024-03-29T11:50:19.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.216+0000 7f2b21424640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:19.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.216+0000 7f2b21424640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:50:19.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.217+0000 7f2b21424640 10 monclient(hunting): picked mon.noname-b con 0x7f2b140389e0 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:19.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.217+0000 7f2b21424640 10 monclient(hunting): picked mon.noname-c con 0x7f2b14038dc0 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:19.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.217+0000 7f2b21424640 10 monclient(hunting): picked mon.noname-a con 0x7f2b14030660 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:19.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.217+0000 7f2b21424640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140c54c0 con 0x7f2b14030660 2024-03-29T11:50:19.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.217+0000 7f2b21424640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b14036430 con 0x7f2b14038dc0 2024-03-29T11:50:19.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.217+0000 7f2b21424640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b14039840 con 0x7f2b140389e0 2024-03-29T11:50:19.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.217+0000 7f2b21424640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:19.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.217+0000 7f2b21424640 10 monclient(hunting): authenticate will time out at 1432.495715s 2024-03-29T11:50:19.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.219+0000 7f2b1bfff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f2b140389e0 0x7f2b14033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:42276/0 (socket says 172.21.15.55:42276) 2024-03-29T11:50:19.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.219+0000 7f2b1bfff640 1 -- 172.21.15.55:0/122038677 learned_addr learned my addr 172.21.15.55:0/122038677 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:19.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.219+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260359555 0 0) 0x7f2b14039840 con 0x7f2b140389e0 2024-03-29T11:50:19.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.219+0000 7f2b1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:19.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.219+0000 7f2b1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:19.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.219+0000 7f2b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:19.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.219+0000 7f2b1affd640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.219+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b0c003520 con 0x7f2b140389e0 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006972420 0 0) 0x7f2b14036430 con 0x7f2b14038dc0 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b14039840 con 0x7f2b14038dc0 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3350096131 0 0) 0x7f2b0c003520 con 0x7f2b140389e0 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 >> v1:172.21.15.171:6789/0 conn(0x7f2b14038dc0 legacy=0x7f2b140cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 >> v1:172.21.15.55:6789/0 conn(0x7f2b14030660 legacy=0x7f2b140d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient: _finish_hunting 0 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 1 monclient: found mon.noname-b 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14035690 con 0x7f2b140389e0 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient: _finish_auth 0 2024-03-29T11:50:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient: _check_auth_tickets 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b21424640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b21424640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2b10001690 con 0x7f2b140389e0 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b1affd640 10 monclient: dump: 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:19.229 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.220+0000 7f2b21424640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.221+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b10001920 con 0x7f2b140389e0 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.221+0000 7f2b1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.221+0000 7f2b1affd640 1 -- 172.21.15.55:0/122038677 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2b10001cb0 con 0x7f2b140389e0 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.221+0000 7f2b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.221+0000 7f2b1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.221+0000 7f2b1affd640 10 monclient: dump: 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:19.230 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:19.231 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.221+0000 7f2b21424640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.221+0000 7f2b21424640 4 set_mon_vals no callback set 2024-03-29T11:50:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.222+0000 7f2b21424640 10 monclient: shutdown 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.222+0000 7f2b21424640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.222+0000 7f2b21424640 1 -- 172.21.15.55:0/122038677 >> v1:172.21.15.55:6790/0 conn(0x7f2b140389e0 legacy=0x7f2b14033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.222+0000 7f2b21424640 1 -- 172.21.15.55:0/122038677 shutdown_connections 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.222+0000 7f2b21424640 1 -- 172.21.15.55:0/122038677 >> 172.21.15.55:0/122038677 conn(0x7f2b14005c00 msgr2=0x7f2b14006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.222+0000 7f2b21424640 1 -- 172.21.15.55:0/122038677 shutdown_connections 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.223+0000 7f2b21424640 1 -- 172.21.15.55:0/122038677 wait complete. 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.223+0000 7f2b21424640 10 monclient: build_initial_monmap 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.223+0000 7f2b21424640 10 monclient: monmap: 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:19.232 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.224+0000 7f2b21424640 1 librados: starting msgr at 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.224+0000 7f2b21424640 1 librados: starting objecter 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.224+0000 7f2b21424640 1 Processor -- start 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.224+0000 7f2b21424640 1 -- start start 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.224+0000 7f2b21424640 1 librados: setting wanted keys 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.224+0000 7f2b21424640 1 librados: calling monclient init 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.224+0000 7f2b21424640 10 monclient: init 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 10 monclient(hunting): picked mon.noname-a con 0x7f2b14030660 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 10 monclient(hunting): picked mon.noname-c con 0x7f2b140389e0 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 10 monclient(hunting): picked mon.noname-b con 0x7f2b1402ef90 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1402f590 con 0x7f2b14030660 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140bfc50 con 0x7f2b1402ef90 2024-03-29T11:50:19.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140c5730 con 0x7f2b140389e0 2024-03-29T11:50:19.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:19.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b21424640 10 monclient(hunting): authenticate will time out at 1432.504271s 2024-03-29T11:50:19.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b1b7fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f2b140389e0 0x7f2b140bee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:46332/0 (socket says 172.21.15.55:46332) 2024-03-29T11:50:19.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.225+0000 7f2b1b7fe640 1 -- 172.21.15.55:0/1301638400 learned_addr learned my addr 172.21.15.55:0/1301638400 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:19.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.226+0000 7f2b18ff9640 10 client.?.objecter ms_handle_connect 0x7f2b140389e0 2024-03-29T11:50:19.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.226+0000 7f2b18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:19.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.226+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668212294 0 0) 0x7f2b140c5730 con 0x7f2b140389e0 2024-03-29T11:50:19.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.226+0000 7f2b18ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:50:19.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.226+0000 7f2b18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:19.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.226+0000 7f2b18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:19.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.226+0000 7f2b18ff9640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:50:19.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.227+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b00004eb0 con 0x7f2b140389e0 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.229+0000 7f2b18ff9640 10 client.?.objecter ms_handle_connect 0x7f2b14030660 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.229+0000 7f2b18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.229+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 711872765 0 0) 0x7f2b00004eb0 con 0x7f2b140389e0 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.230+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b140c5730 con 0x7f2b140389e0 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.230+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2b1c04fdd0 con 0x7f2b140389e0 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.230+0000 7f2b18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.230+0000 7f2b18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.230+0000 7f2b18ff9640 10 monclient(hunting): dump: 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:19.236 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:19.237 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:19.237 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:19.237 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:19.237 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:19.237 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:19.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.230+0000 7f2b18ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:19.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.230+0000 7f2b18ff9640 10 client.?.objecter ms_handle_connect 0x7f2b1402ef90 2024-03-29T11:50:19.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.230+0000 7f2b18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:19.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349209124 0 0) 0x7f2b1402f590 con 0x7f2b14030660 2024-03-29T11:50:19.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b00004eb0 con 0x7f2b14030660 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2779119281 0 0) 0x7f2b140c5730 con 0x7f2b140389e0 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 >> v1:172.21.15.171:6789/0 conn(0x7f2b1402ef90 legacy=0x7f2b140bf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 >> v1:172.21.15.55:6789/0 conn(0x7f2b14030660 legacy=0x7f2b1417dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 1 monclient: found mon.c 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1402ea40 con 0x7f2b140389e0 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 10 monclient: _finish_auth 0 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:50:19.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b21424640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b21424640 10 monclient: _renew_subs 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b21424640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b21424640 1 -- 172.21.15.55:0/1301638400 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1402ec30 con 0x7f2b140389e0 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b21424640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b21424640 10 monclient: _renew_subs 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b21424640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b21424640 1 -- 172.21.15.55:0/1301638400 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b140c1be0 con 0x7f2b140389e0 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.231+0000 7f2b21424640 1 librados: init done 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.232+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b1c04f400 con 0x7f2b140389e0 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.232+0000 7f2b18ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.232+0000 7f2b19ffb640 4 set_mon_vals no callback set 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.233+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2b1c075710 con 0x7f2b140389e0 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.233+0000 7f2b18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.233+0000 7f2b18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.233+0000 7f2b18ff9640 10 monclient: dump: 2024-03-29T11:50:19.240 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:19.241 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:19.241 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:19.241 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:19.241 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:19.241 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:19.241 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:19.241 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:19.241 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:19.241 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:19.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.233+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f2b1c067bf0 con 0x7f2b140389e0 2024-03-29T11:50:19.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.234+0000 7f2b067fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:19.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.234+0000 7f2b067fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:19.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.234+0000 7f2b067fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:19.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.234+0000 7f2b067fc640 1 -- 172.21.15.55:0/1301638400 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b14007b90 con 0x7f2b140389e0 2024-03-29T11:50:19.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.234+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f2b1c0ff630 con 0x7f2b140389e0 2024-03-29T11:50:19.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.234+0000 7f2b18ff9640 10 client.4538.objecter ms_dispatch 0x7f2b1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:19.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.234+0000 7f2b18ff9640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:19.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.234+0000 7f2b18ff9640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:19.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.234+0000 7f2b18ff9640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T11:50:19.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.235+0000 7f2b1bfff640 10 monclient: get_auth_request con 0x7f2b0006db60 auth_method 0 2024-03-29T11:50:19.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.235+0000 7f2b18ff9640 10 client.4538.objecter ms_handle_connect 0x7f2b0006db60 2024-03-29T11:50:19.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.243+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b1c0cb020 con 0x7f2b140389e0 2024-03-29T11:50:19.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.243+0000 7f2b18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:19.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.243+0000 7f2b18ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:19.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.583+0000 7f2b067fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:19.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.583+0000 7f2b067fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:19.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.583+0000 7f2b067fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:19.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.583+0000 7f2b067fc640 1 -- 172.21.15.55:0/1301638400 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b1402edc0 con 0x7f2b140389e0 2024-03-29T11:50:19.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.584+0000 7f2b18ff9640 1 -- 172.21.15.55:0/1301638400 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f2b1c0ccf50 con 0x7f2b140389e0 2024-03-29T11:50:19.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.584+0000 7f2b18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:19.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.584+0000 7f2b18ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:19.590 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:50:19.590 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:19.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.587+0000 7f2b067fc640 10 librados: watch_flush enter 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.588+0000 7f2b067fc640 10 librados: watch_flush exit 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 10 client.4538.objecter successfully canceled tick 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 1 -- 172.21.15.55:0/1301638400 >> v1:172.21.15.171:6816/1463197163 conn(0x7f2b0006db60 legacy=0x7f2b00070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 10 monclient: shutdown 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 1 -- 172.21.15.55:0/1301638400 >> v1:172.21.15.55:6790/0 conn(0x7f2b140389e0 legacy=0x7f2b140bee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 1 -- 172.21.15.55:0/1301638400 shutdown_connections 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 1 -- 172.21.15.55:0/1301638400 >> 172.21.15.55:0/1301638400 conn(0x7f2b14005c00 msgr2=0x7f2b140c2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:19.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 1 -- 172.21.15.55:0/1301638400 shutdown_connections 2024-03-29T11:50:19.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 1 -- 172.21.15.55:0/1301638400 wait complete. 2024-03-29T11:50:19.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:19.589+0000 7f2b067fc640 1 librados: shutdown 2024-03-29T11:50:19.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:26.417 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:26.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.563+0000 7fd6854ae640 10 monclient: get_monmap_and_config 2024-03-29T11:50:26.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.563+0000 7fd6854ae640 10 monclient: build_initial_monmap 2024-03-29T11:50:26.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.563+0000 7fd6854ae640 10 monclient: monmap: 2024-03-29T11:50:26.570 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:26.571 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:26.571 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:50:26.564634+0000 2024-03-29T11:50:26.571 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:50:26.564634+0000 2024-03-29T11:50:26.571 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:26.571 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:26.571 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:26.571 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:50:26.571 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:50:26.571 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:26.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.567+0000 7fd6854ae640 1 Processor -- start 2024-03-29T11:50:26.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.568+0000 7fd6854ae640 1 -- start start 2024-03-29T11:50:26.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.568+0000 7fd6854ae640 10 monclient: init 2024-03-29T11:50:26.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.569+0000 7fd6854ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:26.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.569+0000 7fd6854ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:50:26.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.569+0000 7fd6854ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd6800a9ba0 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:26.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.569+0000 7fd6854ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd68013b5a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:26.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.569+0000 7fd6854ae640 10 monclient(hunting): picked mon.noname-b con 0x7fd6800ab550 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:26.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.571+0000 7fd6854ae640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800ac430 con 0x7fd6800a9ba0 2024-03-29T11:50:26.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.571+0000 7fd6854ae640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800ac680 con 0x7fd68013b5a0 2024-03-29T11:50:26.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.571+0000 7fd6854ae640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800ac960 con 0x7fd6800ab550 2024-03-29T11:50:26.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.571+0000 7fd6854ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:26.578 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.571+0000 7fd6854ae640 10 monclient(hunting): authenticate will time out at 1439.851040s 2024-03-29T11:50:26.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.572+0000 7fd67effd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fd6800a9ba0 0x7fd68013ae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46533/0 (socket says 172.21.15.55:46533) 2024-03-29T11:50:26.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.572+0000 7fd67effd640 1 -- 172.21.15.55:0/780421879 learned_addr learned my addr 172.21.15.55:0/780421879 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:26.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.573+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288831830 0 0) 0x7fd6800ac430 con 0x7fd6800a9ba0 2024-03-29T11:50:26.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.573+0000 7fd67dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:50:26.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.573+0000 7fd67dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:26.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.573+0000 7fd67dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:26.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.574+0000 7fd67dffb640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:50:26.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.574+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd668003520 con 0x7fd6800a9ba0 2024-03-29T11:50:26.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.574+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114289533 0 0) 0x7fd6800ac680 con 0x7fd68013b5a0 2024-03-29T11:50:26.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.574+0000 7fd67dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:50:26.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.574+0000 7fd67dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:26.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.574+0000 7fd67dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:26.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.574+0000 7fd67dffb640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:50:26.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.574+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6800ac430 con 0x7fd68013b5a0 2024-03-29T11:50:26.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 729276756 0 0) 0x7fd668003520 con 0x7fd6800a9ba0 2024-03-29T11:50:26.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 >> v1:172.21.15.55:6790/0 conn(0x7fd6800ab550 legacy=0x7fd6801498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:50:26.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 >> v1:172.21.15.171:6789/0 conn(0x7fd68013b5a0 legacy=0x7fd6800aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:26.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:50:26.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 1 monclient: found mon.noname-a 2024-03-29T11:50:26.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:26.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:50:26.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68013baf0 con 0x7fd6800a9ba0 2024-03-29T11:50:26.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 10 monclient: _finish_auth 0 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd670001690 con 0x7fd6800a9ba0 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.576+0000 7fd67dffb640 10 monclient: dump: 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:26.584 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:26.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.577+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6700018b0 con 0x7fd6800a9ba0 2024-03-29T11:50:26.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.577+0000 7fd67dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:26.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.577+0000 7fd67dffb640 1 -- 172.21.15.55:0/780421879 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd670001c40 con 0x7fd6800a9ba0 2024-03-29T11:50:26.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.577+0000 7fd67dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:26.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.577+0000 7fd67dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:26.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.577+0000 7fd67dffb640 10 monclient: dump: 2024-03-29T11:50:26.585 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.577+0000 7fd6854ae640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.577+0000 7fd6854ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.577+0000 7fd6854ae640 4 set_mon_vals no callback set 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.578+0000 7fd6854ae640 10 monclient: shutdown 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.578+0000 7fd6854ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:26.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.578+0000 7fd6854ae640 1 -- 172.21.15.55:0/780421879 >> v1:172.21.15.55:6789/0 conn(0x7fd6800a9ba0 legacy=0x7fd68013ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:26.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.581+0000 7fd6854ae640 1 -- 172.21.15.55:0/780421879 shutdown_connections 2024-03-29T11:50:26.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.581+0000 7fd6854ae640 1 -- 172.21.15.55:0/780421879 >> 172.21.15.55:0/780421879 conn(0x7fd6800b3a00 msgr2=0x7fd6800b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:26.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.582+0000 7fd6854ae640 1 -- 172.21.15.55:0/780421879 shutdown_connections 2024-03-29T11:50:26.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.582+0000 7fd6854ae640 1 -- 172.21.15.55:0/780421879 wait complete. 2024-03-29T11:50:26.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.583+0000 7fd6854ae640 10 monclient: build_initial_monmap 2024-03-29T11:50:26.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.583+0000 7fd6854ae640 10 monclient: monmap: 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:26.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.584+0000 7fd6854ae640 1 librados: starting msgr at 2024-03-29T11:50:26.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.584+0000 7fd6854ae640 1 librados: starting objecter 2024-03-29T11:50:26.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.585+0000 7fd6854ae640 1 Processor -- start 2024-03-29T11:50:26.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.585+0000 7fd6854ae640 1 -- start start 2024-03-29T11:50:26.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.585+0000 7fd6854ae640 1 librados: setting wanted keys 2024-03-29T11:50:26.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.585+0000 7fd6854ae640 1 librados: calling monclient init 2024-03-29T11:50:26.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.585+0000 7fd6854ae640 10 monclient: init 2024-03-29T11:50:26.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.587+0000 7fd6854ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:26.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.587+0000 7fd6854ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:50:26.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.587+0000 7fd6854ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd6800a9ba0 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:26.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.587+0000 7fd6854ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd6800ab550 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:26.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.591+0000 7fd6854ae640 10 monclient(hunting): picked mon.noname-b con 0x7fd68013b5a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:26.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.591+0000 7fd6854ae640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800727c0 con 0x7fd6800ab550 2024-03-29T11:50:26.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.591+0000 7fd6854ae640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68006dc70 con 0x7fd68013b5a0 2024-03-29T11:50:26.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.591+0000 7fd6854ae640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68006de00 con 0x7fd6800a9ba0 2024-03-29T11:50:26.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.591+0000 7fd6854ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:26.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.591+0000 7fd6854ae640 10 monclient(hunting): authenticate will time out at 1439.870751s 2024-03-29T11:50:26.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.593+0000 7fd67f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd68013b5a0 0x7fd6801d6090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:57591/0 (socket says 172.21.15.55:57591) 2024-03-29T11:50:26.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.593+0000 7fd67f7fe640 1 -- 172.21.15.55:0/1185027922 learned_addr learned my addr 172.21.15.55:0/1185027922 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:26.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.593+0000 7fd6677fe640 10 client.?.objecter ms_handle_connect 0x7fd68013b5a0 2024-03-29T11:50:26.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.593+0000 7fd6677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:26.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.594+0000 7fd6677fe640 10 client.?.objecter ms_handle_connect 0x7fd6800ab550 2024-03-29T11:50:26.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.594+0000 7fd6677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:26.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.594+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959796210 0 0) 0x7fd6800727c0 con 0x7fd6800ab550 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.594+0000 7fd6677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.594+0000 7fd6677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.594+0000 7fd6677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.594+0000 7fd6677fe640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.594+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd66c004eb0 con 0x7fd6800ab550 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.595+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1280763939 0 0) 0x7fd66c004eb0 con 0x7fd6800ab550 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.595+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6800727c0 con 0x7fd6800ab550 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.595+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd674001910 con 0x7fd6800ab550 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.595+0000 7fd6677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.595+0000 7fd6677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.595+0000 7fd6677fe640 10 monclient(hunting): dump: 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:26.605 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.595+0000 7fd6677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3872124134 0 0) 0x7fd6800727c0 con 0x7fd6800ab550 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 >> v1:172.21.15.55:6790/0 conn(0x7fd6800a9ba0 legacy=0x7fd68006e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 >> v1:172.21.15.171:6789/0 conn(0x7fd68013b5a0 legacy=0x7fd6801d6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 1 monclient: found mon.a 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:26.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68006ef60 con 0x7fd6800ab550 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 10 monclient: _finish_auth 0 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6854ae640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd674003610 con 0x7fd6800ab550 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd674003920 con 0x7fd6800ab550 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.596+0000 7fd6677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.597+0000 7fd6677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.597+0000 7fd6677fe640 10 monclient: dump: 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:26.607 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:26.608 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:26.608 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:26.608 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:26.608 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:26.608 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:26.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.597+0000 7fd6854ae640 10 monclient: _renew_subs 2024-03-29T11:50:26.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.597+0000 7fd6854ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:26.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.597+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68006f150 con 0x7fd6800ab550 2024-03-29T11:50:26.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.597+0000 7fd6854ae640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:26.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.598+0000 7fd6854ae640 10 monclient: _renew_subs 2024-03-29T11:50:26.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.598+0000 7fd6854ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:26.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.598+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd680072320 con 0x7fd6800ab550 2024-03-29T11:50:26.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.598+0000 7fd6854ae640 1 librados: init done 2024-03-29T11:50:26.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.597+0000 7fd67d7fa640 4 set_mon_vals no callback set 2024-03-29T11:50:26.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.600+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fd6740198f0 con 0x7fd6800ab550 2024-03-29T11:50:26.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.600+0000 7fd6854ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:26.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.601+0000 7fd6854ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:26.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.601+0000 7fd6854ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:26.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.601+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6800725d0 con 0x7fd6800ab550 2024-03-29T11:50:26.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.601+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fd6740536b0 con 0x7fd6800ab550 2024-03-29T11:50:26.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.601+0000 7fd6677fe640 10 client.4533.objecter ms_dispatch 0x7fd680000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:26.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.601+0000 7fd6677fe640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:26.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.601+0000 7fd6677fe640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:26.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.601+0000 7fd6677fe640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T11:50:26.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.605+0000 7fd67effd640 10 monclient: get_auth_request con 0x7fd66c06db80 auth_method 0 2024-03-29T11:50:26.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.605+0000 7fd6677fe640 10 client.4533.objecter ms_handle_connect 0x7fd66c06db80 2024-03-29T11:50:26.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.609+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd67404f1c0 con 0x7fd6800ab550 2024-03-29T11:50:26.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.609+0000 7fd6677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:26.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.609+0000 7fd6677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:26.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.944+0000 7fd6854ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:26.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.944+0000 7fd6854ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:26.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.944+0000 7fd6854ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:26.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.944+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd68004f730 con 0x7fd6800ab550 2024-03-29T11:50:26.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.946+0000 7fd6677fe640 1 -- 172.21.15.55:0/1185027922 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fd6740510f0 con 0x7fd6800ab550 2024-03-29T11:50:26.953 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:50:26.953 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:26.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.946+0000 7fd6677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:26.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.946+0000 7fd6677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:26.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.949+0000 7fd6854ae640 10 librados: watch_flush enter 2024-03-29T11:50:26.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.950+0000 7fd6854ae640 10 librados: watch_flush exit 2024-03-29T11:50:26.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.950+0000 7fd6854ae640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T11:50:26.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.950+0000 7fd6854ae640 10 client.4533.objecter successfully canceled tick 2024-03-29T11:50:26.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.950+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 >> v1:172.21.15.171:6816/1463197163 conn(0x7fd66c06db80 legacy=0x7fd66c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:26.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.950+0000 7fd6854ae640 10 monclient: shutdown 2024-03-29T11:50:26.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.950+0000 7fd6854ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:26.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.950+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 >> v1:172.21.15.55:6789/0 conn(0x7fd6800ab550 legacy=0x7fd680071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:26.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.953+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 shutdown_connections 2024-03-29T11:50:26.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.953+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 >> 172.21.15.55:0/1185027922 conn(0x7fd6800b3a00 msgr2=0x7fd680069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:26.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.953+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 shutdown_connections 2024-03-29T11:50:26.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.953+0000 7fd6854ae640 1 -- 172.21.15.55:0/1185027922 wait complete. 2024-03-29T11:50:26.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:26.955+0000 7fd6854ae640 1 librados: shutdown 2024-03-29T11:50:26.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:33.787 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:33.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.932+0000 7f5484d96640 10 monclient: get_monmap_and_config 2024-03-29T11:50:33.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.932+0000 7f5484d96640 10 monclient: build_initial_monmap 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.932+0000 7f5484d96640 10 monclient: monmap: 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:50:33.933777+0000 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:50:33.933777+0000 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:50:33.940 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:33.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.935+0000 7f5484d96640 1 Processor -- start 2024-03-29T11:50:33.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.935+0000 7f5484d96640 1 -- start start 2024-03-29T11:50:33.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.935+0000 7f5484d96640 10 monclient: init 2024-03-29T11:50:33.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.936+0000 7f5484d96640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.937+0000 7f5484d96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.937+0000 7f5484d96640 10 monclient(hunting): picked mon.noname-c con 0x7f54800b0770 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.937+0000 7f5484d96640 10 monclient(hunting): picked mon.noname-a con 0x7f548014a700 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.937+0000 7f5484d96640 10 monclient(hunting): picked mon.noname-b con 0x7f54800ac5a0 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.937+0000 7f5484d96640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54800aba70 con 0x7f548014a700 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.937+0000 7f5484d96640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54800aecb0 con 0x7f54800b0770 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.937+0000 7f5484d96640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54800af150 con 0x7f54800ac5a0 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.937+0000 7f5484d96640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.937+0000 7f5484d96640 10 monclient(hunting): authenticate will time out at 1447.217504s 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.938+0000 7f547ed9d640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f54800ac5a0 0x7f5480151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:40018/0 (socket says 172.21.15.55:40018) 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.938+0000 7f547ed9d640 1 -- 172.21.15.55:0/3502796959 learned_addr learned my addr 172.21.15.55:0/3502796959 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.938+0000 7f547d59a640 1 -- 172.21.15.55:0/3502796959 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294758094 0 0) 0x7f54800af150 con 0x7f54800ac5a0 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.938+0000 7f547d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:50:33.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.938+0000 7f547d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.938+0000 7f547d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.938+0000 7f547d59a640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.938+0000 7f547d59a640 1 -- 172.21.15.55:0/3502796959 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5468003520 con 0x7f54800ac5a0 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.939+0000 7f547d59a640 1 -- 172.21.15.55:0/3502796959 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3016408768 0 0) 0x7f5468003520 con 0x7f54800ac5a0 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.939+0000 7f547d59a640 1 -- 172.21.15.55:0/3502796959 >> v1:172.21.15.171:6789/0 conn(0x7f54800b0770 legacy=0x7f5480149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 1 -- 172.21.15.55:0/3502796959 >> v1:172.21.15.55:6789/0 conn(0x7f548014a700 legacy=0x7f54800abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 1 monclient: found mon.noname-b 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 1 -- 172.21.15.55:0/3502796959 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54800ac9d0 con 0x7f54800ac5a0 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 10 monclient: _finish_auth 0 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:50:33.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 1 -- 172.21.15.55:0/3502796959 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5478001a10 con 0x7f54800ac5a0 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.940+0000 7f547d59a640 10 monclient: dump: 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:33.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.941+0000 7f5484d96640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.941+0000 7f5484d96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.942+0000 7f547d59a640 1 -- 172.21.15.55:0/3502796959 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5478001be0 con 0x7f54800ac5a0 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.942+0000 7f547d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.942+0000 7f547d59a640 1 -- 172.21.15.55:0/3502796959 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5478002030 con 0x7f54800ac5a0 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.942+0000 7f547d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.942+0000 7f547d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.942+0000 7f547d59a640 10 monclient: dump: 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:33.955 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:33.956 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:33.956 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:33.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.942+0000 7f5484d96640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:33.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.942+0000 7f5484d96640 4 set_mon_vals no callback set 2024-03-29T11:50:33.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.943+0000 7f5484d96640 10 monclient: shutdown 2024-03-29T11:50:33.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.943+0000 7f5484d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:33.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.943+0000 7f5484d96640 1 -- 172.21.15.55:0/3502796959 >> v1:172.21.15.55:6790/0 conn(0x7f54800ac5a0 legacy=0x7f5480151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.946+0000 7f5484d96640 1 -- 172.21.15.55:0/3502796959 shutdown_connections 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.946+0000 7f5484d96640 1 -- 172.21.15.55:0/3502796959 >> 172.21.15.55:0/3502796959 conn(0x7f548006b660 msgr2=0x7f548006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.946+0000 7f5484d96640 1 -- 172.21.15.55:0/3502796959 shutdown_connections 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.947+0000 7f5484d96640 1 -- 172.21.15.55:0/3502796959 wait complete. 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.948+0000 7f5484d96640 10 monclient: build_initial_monmap 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.948+0000 7f5484d96640 10 monclient: monmap: 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:33.958 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.948+0000 7f5484d96640 1 librados: starting msgr at 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.948+0000 7f5484d96640 1 librados: starting objecter 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.948+0000 7f5484d96640 1 Processor -- start 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.948+0000 7f5484d96640 1 -- start start 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.948+0000 7f5484d96640 1 librados: setting wanted keys 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.948+0000 7f5484d96640 1 librados: calling monclient init 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.948+0000 7f5484d96640 10 monclient: init 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.949+0000 7f5484d96640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.949+0000 7f5484d96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.950+0000 7f5484d96640 10 monclient(hunting): picked mon.noname-a con 0x7f54800ac5a0 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:33.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.950+0000 7f5484d96640 10 monclient(hunting): picked mon.noname-b con 0x7f54800b0770 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:33.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.950+0000 7f5484d96640 10 monclient(hunting): picked mon.noname-c con 0x7f548014a700 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:33.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.950+0000 7f5484d96640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54801285d0 con 0x7f54800ac5a0 2024-03-29T11:50:33.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.951+0000 7f5484d96640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5480129090 con 0x7f54800b0770 2024-03-29T11:50:33.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.951+0000 7f5484d96640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5480129220 con 0x7f548014a700 2024-03-29T11:50:33.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.951+0000 7f5484d96640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:33.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.951+0000 7f5484d96640 10 monclient(hunting): authenticate will time out at 1447.230942s 2024-03-29T11:50:33.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f547ed9d640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f548014a700 0x7f54801e6b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:47162/0 (socket says 172.21.15.55:47162) 2024-03-29T11:50:33.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f547ed9d640 1 -- 172.21.15.55:0/1121333560 learned_addr learned my addr 172.21.15.55:0/1121333560 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:33.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f546effd640 10 client.?.objecter ms_handle_connect 0x7f548014a700 2024-03-29T11:50:33.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f546effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:33.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603026418 0 0) 0x7f5480129220 con 0x7f548014a700 2024-03-29T11:50:33.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f546effd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:33.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f546effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:33.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f546effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:33.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f546effd640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:50:33.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.952+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5460004eb0 con 0x7f548014a700 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 854366982 0 0) 0x7f5460004eb0 con 0x7f548014a700 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5480129220 con 0x7f548014a700 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5478001ef0 con 0x7f548014a700 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient(hunting): dump: 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:33.964 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 client.?.objecter ms_handle_connect 0x7f54800ac5a0 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520568048 0 0) 0x7f54801285d0 con 0x7f54800ac5a0 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5460004eb0 con 0x7f54800ac5a0 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2738517345 0 0) 0x7f5480129220 con 0x7f548014a700 2024-03-29T11:50:33.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 >> v1:172.21.15.171:6789/0 conn(0x7f54800b0770 legacy=0x7f54801deb40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 >> v1:172.21.15.55:6789/0 conn(0x7f54800ac5a0 legacy=0x7f5480146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient: _finish_hunting 0 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 monclient: found mon.c 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54801475b0 con 0x7f548014a700 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient: _finish_auth 0 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 10 monclient: _check_auth_tickets 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.955+0000 7f546effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f5484d96640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f5484d96640 10 monclient: _renew_subs 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5478004f70 con 0x7f548014a700 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f5484d96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54801477a0 con 0x7f548014a700 2024-03-29T11:50:33.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f546effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5478005280 con 0x7f548014a700 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f546effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f546effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f546effd640 10 monclient: dump: 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:33.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f5484d96640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:33.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f5484d96640 10 monclient: _renew_subs 2024-03-29T11:50:33.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f5484d96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:33.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5480147c00 con 0x7f548014a700 2024-03-29T11:50:33.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.956+0000 7f5484d96640 1 librados: init done 2024-03-29T11:50:33.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.957+0000 7f547cd99640 4 set_mon_vals no callback set 2024-03-29T11:50:33.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.960+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f547801b2d0 con 0x7f548014a700 2024-03-29T11:50:33.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.961+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f5478085050 con 0x7f548014a700 2024-03-29T11:50:33.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.961+0000 7f546effd640 10 client.4544.objecter ms_dispatch 0x7f5480000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:33.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.961+0000 7f546effd640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:33.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.961+0000 7f546effd640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:33.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.961+0000 7f546effd640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T11:50:33.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.961+0000 7f547e59c640 10 monclient: get_auth_request con 0x7f546006dcc0 auth_method 0 2024-03-29T11:50:33.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.962+0000 7f5484d96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:33.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.962+0000 7f5484d96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:33.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.962+0000 7f5484d96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:33.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.962+0000 7f546effd640 10 client.4544.objecter ms_handle_connect 0x7f546006dcc0 2024-03-29T11:50:33.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.964+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5464005080 con 0x7f548014a700 2024-03-29T11:50:33.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.975+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5478050930 con 0x7f548014a700 2024-03-29T11:50:33.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.975+0000 7f546effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:33.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:33.975+0000 7f546effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:34.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.309+0000 7f5484d96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:34.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.309+0000 7f5484d96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:34.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.309+0000 7f5484d96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:34.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.309+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5464005330 con 0x7f548014a700 2024-03-29T11:50:34.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.311+0000 7f546effd640 1 -- 172.21.15.55:0/1121333560 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f5478052860 con 0x7f548014a700 2024-03-29T11:50:34.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.311+0000 7f546effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:34.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.311+0000 7f546effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:34.319 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:50:34.319 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:34.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.315+0000 7f5484d96640 10 librados: watch_flush enter 2024-03-29T11:50:34.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.315+0000 7f5484d96640 10 librados: watch_flush exit 2024-03-29T11:50:34.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.315+0000 7f5484d96640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T11:50:34.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.315+0000 7f5484d96640 10 client.4544.objecter successfully canceled tick 2024-03-29T11:50:34.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.316+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 >> v1:172.21.15.171:6816/1463197163 conn(0x7f546006dcc0 legacy=0x7f5460070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:34.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.316+0000 7f5484d96640 10 monclient: shutdown 2024-03-29T11:50:34.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.316+0000 7f5484d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:34.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.316+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 >> v1:172.21.15.55:6790/0 conn(0x7f548014a700 legacy=0x7f54801e6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:34.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.317+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 shutdown_connections 2024-03-29T11:50:34.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.317+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 >> 172.21.15.55:0/1121333560 conn(0x7f548006b660 msgr2=0x7f54800b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:34.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.317+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 shutdown_connections 2024-03-29T11:50:34.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.318+0000 7f5484d96640 1 -- 172.21.15.55:0/1121333560 wait complete. 2024-03-29T11:50:34.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:34.318+0000 7f5484d96640 1 librados: shutdown 2024-03-29T11:50:34.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:41.152 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.298+0000 7f981f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.298+0000 7f981f59e640 10 monclient: build_initial_monmap 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.298+0000 7f981f59e640 10 monclient: monmap: 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:50:41.299441+0000 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:50:41.299441+0000 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:41.307 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.299+0000 7f981f59e640 1 Processor -- start 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.299+0000 7f981f59e640 1 -- start start 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.299+0000 7f981f59e640 10 monclient: init 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f981802b090 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f9818029e10 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f98180e2c40 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981802a250 con 0x7f98180e2c40 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981802c0f0 con 0x7f9818029e10 2024-03-29T11:50:41.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981802a430 con 0x7f981802b090 2024-03-29T11:50:41.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:41.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.300+0000 7f981f59e640 10 monclient(hunting): authenticate will time out at 1454.580212s 2024-03-29T11:50:41.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.301+0000 7f981e59c640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f981802b090 0x7f98180e8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:33052/0 (socket says 172.21.15.55:33052) 2024-03-29T11:50:41.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.301+0000 7f981e59c640 1 -- 172.21.15.55:0/1708439062 learned_addr learned my addr 172.21.15.55:0/1708439062 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:41.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 1 -- 172.21.15.55:0/1708439062 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029759260 0 0) 0x7f981802a430 con 0x7f981802b090 2024-03-29T11:50:41.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:50:41.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:41.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:41.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:50:41.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 1 -- 172.21.15.55:0/1708439062 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9808003520 con 0x7f981802b090 2024-03-29T11:50:41.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 1 -- 172.21.15.55:0/1708439062 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494732359 0 0) 0x7f9808003520 con 0x7f981802b090 2024-03-29T11:50:41.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 1 -- 172.21.15.55:0/1708439062 >> v1:172.21.15.171:6789/0 conn(0x7f9818029e10 legacy=0x7f98180e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:41.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 1 -- 172.21.15.55:0/1708439062 >> v1:172.21.15.55:6789/0 conn(0x7f98180e2c40 legacy=0x7f98180e9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:41.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:50:41.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 1 monclient: found mon.noname-b 2024-03-29T11:50:41.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 1 -- 172.21.15.55:0/1708439062 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981802a5c0 con 0x7f981802b090 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient: _finish_auth 0 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 1 -- 172.21.15.55:0/1708439062 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9814001690 con 0x7f981802b090 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981d59a640 10 monclient: dump: 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:41.313 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981f59e640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.304+0000 7f981f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.305+0000 7f981d59a640 1 -- 172.21.15.55:0/1708439062 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9814001920 con 0x7f981802b090 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.305+0000 7f981d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.305+0000 7f981d59a640 1 -- 172.21.15.55:0/1708439062 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9814001cb0 con 0x7f981802b090 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.305+0000 7f981d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.305+0000 7f981d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.305+0000 7f981d59a640 10 monclient: dump: 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.314 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:41.315 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:41.315 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:41.315 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:41.315 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:41.315 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:41.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.305+0000 7f981f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:41.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.305+0000 7f981f59e640 4 set_mon_vals no callback set 2024-03-29T11:50:41.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.306+0000 7f981f59e640 10 monclient: shutdown 2024-03-29T11:50:41.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.306+0000 7f981f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.306+0000 7f981f59e640 1 -- 172.21.15.55:0/1708439062 >> v1:172.21.15.55:6790/0 conn(0x7f981802b090 legacy=0x7f98180e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.306+0000 7f981f59e640 1 -- 172.21.15.55:0/1708439062 shutdown_connections 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.306+0000 7f981f59e640 1 -- 172.21.15.55:0/1708439062 >> 172.21.15.55:0/1708439062 conn(0x7f98180058f0 msgr2=0x7f9818005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.306+0000 7f981f59e640 1 -- 172.21.15.55:0/1708439062 shutdown_connections 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.306+0000 7f981f59e640 1 -- 172.21.15.55:0/1708439062 wait complete. 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.307+0000 7f981f59e640 10 monclient: build_initial_monmap 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.307+0000 7f981f59e640 10 monclient: monmap: 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:41.317 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.308+0000 7f981f59e640 1 librados: starting msgr at 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.308+0000 7f981f59e640 1 librados: starting objecter 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.308+0000 7f981f59e640 1 Processor -- start 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.308+0000 7f981f59e640 1 -- start start 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.308+0000 7f981f59e640 1 librados: setting wanted keys 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.308+0000 7f981f59e640 1 librados: calling monclient init 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.308+0000 7f981f59e640 10 monclient: init 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.309+0000 7f981f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.309+0000 7f981f59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:50:41.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.309+0000 7f981f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f9818029e10 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:41.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.310+0000 7f981f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f981802b090 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:41.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.310+0000 7f981dd9b640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f981802b090 0x7f98180f52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:48108/0 (socket says 172.21.15.55:48108) 2024-03-29T11:50:41.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.313+0000 7f981ed9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f98180e2c40 0x7f98180f5a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54269/0 (socket says 172.21.15.55:54269) 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.313+0000 7f981f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f98180e2c40 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.313+0000 7f981f59e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981802fec0 con 0x7f98180e2c40 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.313+0000 7f981f59e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9818030cf0 con 0x7f9818029e10 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.313+0000 7f981f59e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9818030fd0 con 0x7f981802b090 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.313+0000 7f981f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.313+0000 7f981f59e640 10 monclient(hunting): authenticate will time out at 1454.593032s 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.315+0000 7f981dd9b640 1 -- 172.21.15.55:0/3396098947 learned_addr learned my addr 172.21.15.55:0/3396098947 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.315+0000 7f980effd640 10 client.?.objecter ms_handle_connect 0x7f98180e2c40 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.315+0000 7f980effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 client.?.objecter ms_handle_connect 0x7f981802b090 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599900681 0 0) 0x7f981802fec0 con 0x7f98180e2c40 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:41.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9800004eb0 con 0x7f98180e2c40 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 client.?.objecter ms_handle_connect 0x7f9818029e10 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955822107 0 0) 0x7f9818030cf0 con 0x7f9818029e10 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f981802fec0 con 0x7f9818029e10 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414498110 0 0) 0x7f9818030fd0 con 0x7f981802b090 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.316+0000 7f980effd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.317+0000 7f980effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.317+0000 7f980effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.317+0000 7f980effd640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:50:41.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.317+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9818030cf0 con 0x7f981802b090 2024-03-29T11:50:41.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.317+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35105814 0 0) 0x7f9800004eb0 con 0x7f98180e2c40 2024-03-29T11:50:41.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.317+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9818030fd0 con 0x7f98180e2c40 2024-03-29T11:50:41.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611373029 0 0) 0x7f981802fec0 con 0x7f9818029e10 2024-03-29T11:50:41.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9800004eb0 con 0x7f9818029e10 2024-03-29T11:50:41.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f9810001810 con 0x7f98180e2c40 2024-03-29T11:50:41.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:41.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient(hunting): dump: 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f98140013b0 con 0x7f9818029e10 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:50:41.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient(hunting): dump: 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2828976114 0 0) 0x7f9818030fd0 con 0x7f98180e2c40 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 >> v1:172.21.15.55:6790/0 conn(0x7f981802b090 legacy=0x7f98180f52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 >> v1:172.21.15.171:6789/0 conn(0x7f9818029e10 legacy=0x7f981802f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient: _finish_hunting 0 2024-03-29T11:50:41.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 monclient: found mon.a 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98180312b0 con 0x7f98180e2c40 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient: _finish_auth 0 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient: _check_auth_tickets 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9810001970 con 0x7f98180e2c40 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.318+0000 7f980ffff640 4 set_mon_vals no callback set 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f981f59e640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f981f59e640 10 monclient: _renew_subs 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f981f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981802f110 con 0x7f98180e2c40 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f981f59e640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:41.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f981f59e640 10 monclient: _renew_subs 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f981f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98180f81e0 con 0x7f98180e2c40 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f981f59e640 1 librados: init done 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f98100037b0 con 0x7f98180e2c40 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f980effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f980effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.321+0000 7f980effd640 10 monclient: dump: 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:41.332 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:41.333 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:41.333 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:41.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.325+0000 7f981f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:41.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.325+0000 7f981f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:41.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.325+0000 7f981f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:41.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.325+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98180e0710 con 0x7f98180e2c40 2024-03-29T11:50:41.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.326+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f9810003910 con 0x7f98180e2c40 2024-03-29T11:50:41.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.327+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f9810084860 con 0x7f98180e2c40 2024-03-29T11:50:41.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.327+0000 7f980effd640 10 client.4542.objecter ms_dispatch 0x7f9820000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:41.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.327+0000 7f980effd640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:41.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.328+0000 7f980effd640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:41.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.328+0000 7f980effd640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:50:41.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.330+0000 7f981e59c640 10 monclient: get_auth_request con 0x7f9800076760 auth_method 0 2024-03-29T11:50:41.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.334+0000 7f980effd640 10 client.4542.objecter ms_handle_connect 0x7f9800076760 2024-03-29T11:50:41.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.334+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9810050140 con 0x7f98180e2c40 2024-03-29T11:50:41.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.334+0000 7f980effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:41.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.334+0000 7f980effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:41.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.669+0000 7f981f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:41.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.669+0000 7f981f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:41.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.669+0000 7f981f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:50:41.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.669+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9818004cd0 con 0x7f98180e2c40 2024-03-29T11:50:41.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.671+0000 7f980effd640 1 -- 172.21.15.55:0/3396098947 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f9810052070 con 0x7f98180e2c40 2024-03-29T11:50:41.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.671+0000 7f980effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:41.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.671+0000 7f980effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:41.679 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:50:41.680 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:41.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.676+0000 7f981f59e640 10 librados: watch_flush enter 2024-03-29T11:50:41.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.676+0000 7f981f59e640 10 librados: watch_flush exit 2024-03-29T11:50:41.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.676+0000 7f981f59e640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:50:41.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.676+0000 7f981f59e640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:50:41.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.676+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 >> v1:172.21.15.171:6816/1463197163 conn(0x7f9800076760 legacy=0x7f9800078c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:41.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.676+0000 7f981f59e640 10 monclient: shutdown 2024-03-29T11:50:41.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.676+0000 7f981f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:41.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.676+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 >> v1:172.21.15.55:6789/0 conn(0x7f98180e2c40 legacy=0x7f98180f5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:41.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.677+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 shutdown_connections 2024-03-29T11:50:41.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.677+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 >> 172.21.15.55:0/3396098947 conn(0x7f98180058f0 msgr2=0x7f98180e4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:41.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.678+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 shutdown_connections 2024-03-29T11:50:41.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.678+0000 7f981f59e640 1 -- 172.21.15.55:0/3396098947 wait complete. 2024-03-29T11:50:41.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:41.680+0000 7f981f59e640 1 librados: shutdown 2024-03-29T11:50:41.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:48.519 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:48.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.671+0000 7f501a7eb640 10 monclient: get_monmap_and_config 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.671+0000 7f501a7eb640 10 monclient: build_initial_monmap 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.671+0000 7f501a7eb640 10 monclient: monmap: 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:50:48.673036+0000 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:50:48.673036+0000 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:50:48.679 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:50:48.680 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:48.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.672+0000 7f501a7eb640 1 Processor -- start 2024-03-29T11:50:48.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.673+0000 7f501a7eb640 1 -- start start 2024-03-29T11:50:48.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.673+0000 7f501a7eb640 10 monclient: init 2024-03-29T11:50:48.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:48.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:50:48.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 10 monclient(hunting): picked mon.noname-a con 0x7f50140ae490 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:48.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 10 monclient(hunting): picked mon.noname-b con 0x7f50140aef70 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:48.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 10 monclient(hunting): picked mon.noname-c con 0x7f50140727a0 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:48.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50140b0020 con 0x7f50140ae490 2024-03-29T11:50:48.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501406d960 con 0x7f50140727a0 2024-03-29T11:50:48.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50140ad310 con 0x7f50140aef70 2024-03-29T11:50:48.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:48.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.674+0000 7f501a7eb640 10 monclient(hunting): authenticate will time out at 1461.953808s 2024-03-29T11:50:48.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.677+0000 7f5019fea640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f50140727a0 0x7f501414b670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:57380/0 (socket says 172.21.15.55:57380) 2024-03-29T11:50:48.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.677+0000 7f5019fea640 1 -- 172.21.15.55:0/3676503173 learned_addr learned my addr 172.21.15.55:0/3676503173 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:48.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.677+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305204662 0 0) 0x7f501406d960 con 0x7f50140727a0 2024-03-29T11:50:48.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 10 monclient(hunting): _negotiate 2024-03-29T11:50:48.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:48.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:48.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:50:48.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc003520 con 0x7f50140727a0 2024-03-29T11:50:48.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20718366 0 0) 0x7f50140b0020 con 0x7f50140ae490 2024-03-29T11:50:48.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 10 monclient(hunting): _negotiate 2024-03-29T11:50:48.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:48.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:48.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:50:48.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f501406d960 con 0x7f50140ae490 2024-03-29T11:50:48.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2600465589 0 0) 0x7f4ffc003520 con 0x7f50140727a0 2024-03-29T11:50:48.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.678+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 >> v1:172.21.15.55:6790/0 conn(0x7f50140aef70 legacy=0x7f5014072090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:50:48.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 >> v1:172.21.15.55:6789/0 conn(0x7f50140ae490 legacy=0x7f50140713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:48.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 10 monclient: _finish_hunting 0 2024-03-29T11:50:48.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 1 monclient: found mon.noname-c 2024-03-29T11:50:48.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:48.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:50:48.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50141432d0 con 0x7f50140727a0 2024-03-29T11:50:48.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 10 monclient: _finish_auth 0 2024-03-29T11:50:48.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 10 monclient: _check_auth_tickets 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5004001990 con 0x7f50140727a0 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.679+0000 7f5013fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.680+0000 7f5013fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.680+0000 7f5013fff640 10 monclient: dump: 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:48.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.680+0000 7f501a7eb640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:50:48.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.680+0000 7f501a7eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:48.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.680+0000 7f5013fff640 1 -- 172.21.15.55:0/3676503173 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5004001bb0 con 0x7f50140727a0 2024-03-29T11:50:48.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.680+0000 7f5013fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:48.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.680+0000 7f501a7eb640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:48.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.680+0000 7f501a7eb640 4 set_mon_vals no callback set 2024-03-29T11:50:48.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.681+0000 7f501a7eb640 10 monclient: shutdown 2024-03-29T11:50:48.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.681+0000 7f501a7eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:48.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.681+0000 7f501a7eb640 1 -- 172.21.15.55:0/3676503173 >> v1:172.21.15.171:6789/0 conn(0x7f50140727a0 legacy=0x7f501414b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:48.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.683+0000 7f501a7eb640 1 -- 172.21.15.55:0/3676503173 shutdown_connections 2024-03-29T11:50:48.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.683+0000 7f501a7eb640 1 -- 172.21.15.55:0/3676503173 >> 172.21.15.55:0/3676503173 conn(0x7f50140673e0 msgr2=0x7f50140b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:48.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.684+0000 7f501a7eb640 1 -- 172.21.15.55:0/3676503173 shutdown_connections 2024-03-29T11:50:48.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.684+0000 7f501a7eb640 1 -- 172.21.15.55:0/3676503173 wait complete. 2024-03-29T11:50:48.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.685+0000 7f501a7eb640 10 monclient: build_initial_monmap 2024-03-29T11:50:48.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.685+0000 7f501a7eb640 10 monclient: monmap: 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:50:48.692 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:48.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.685+0000 7f501a7eb640 1 librados: starting msgr at 2024-03-29T11:50:48.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.685+0000 7f501a7eb640 1 librados: starting objecter 2024-03-29T11:50:48.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.685+0000 7f501a7eb640 1 Processor -- start 2024-03-29T11:50:48.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.685+0000 7f501a7eb640 1 -- start start 2024-03-29T11:50:48.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.685+0000 7f501a7eb640 1 librados: setting wanted keys 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.685+0000 7f501a7eb640 1 librados: calling monclient init 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.685+0000 7f501a7eb640 10 monclient: init 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 10 monclient(hunting): picked mon.noname-a con 0x7f50140727a0 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 10 monclient(hunting): picked mon.noname-b con 0x7f50140ae490 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 10 monclient(hunting): picked mon.noname-c con 0x7f50140aef70 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501411b9c0 con 0x7f50140727a0 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501411c890 con 0x7f50140ae490 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501411cb70 con 0x7f50140aef70 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.686+0000 7f501a7eb640 10 monclient(hunting): authenticate will time out at 1461.966414s 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.687+0000 7f5018fe8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f50140ae490 0x7f501406ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:58204/0 (socket says 172.21.15.55:58204) 2024-03-29T11:50:48.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.687+0000 7f5018fe8640 1 -- 172.21.15.55:0/2687891806 learned_addr learned my addr 172.21.15.55:0/2687891806 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:48.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.688+0000 7f5011ffb640 10 client.?.objecter ms_handle_connect 0x7f50140ae490 2024-03-29T11:50:48.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.688+0000 7f5011ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:48.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.688+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204821088 0 0) 0x7f501411c890 con 0x7f50140ae490 2024-03-29T11:50:48.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.688+0000 7f5011ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.688+0000 7f5011ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.688+0000 7f5011ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.688+0000 7f5011ffb640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.688+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5000004eb0 con 0x7f50140ae490 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.690+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3770142057 0 0) 0x7f5000004eb0 con 0x7f50140ae490 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.690+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f501411c890 con 0x7f50140ae490 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.690+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f50080017d0 con 0x7f50140ae490 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.690+0000 7f5011ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.690+0000 7f5011ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.690+0000 7f5011ffb640 10 monclient(hunting): dump: 2024-03-29T11:50:48.700 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.690+0000 7f5011ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.690+0000 7f5011ffb640 10 client.?.objecter ms_handle_connect 0x7f50140727a0 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 10 client.?.objecter ms_handle_connect 0x7f50140aef70 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3833060376 0 0) 0x7f501411c890 con 0x7f50140ae490 2024-03-29T11:50:48.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 >> v1:172.21.15.55:6790/0 conn(0x7f50140aef70 legacy=0x7f501413bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 >> v1:172.21.15.55:6789/0 conn(0x7f50140727a0 legacy=0x7f50141de9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 1 monclient: found mon.b 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501411b4a0 con 0x7f50140ae490 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 10 monclient: _finish_auth 0 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.691+0000 7f5011ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f501a7eb640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f501a7eb640 10 monclient: _renew_subs 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f501a7eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50141df0b0 con 0x7f50140ae490 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5008001930 con 0x7f50140ae490 2024-03-29T11:50:48.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f5011ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:48.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f501a7eb640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:48.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f501a7eb640 10 monclient: _renew_subs 2024-03-29T11:50:48.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f501a7eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:50:48.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f501411f430 con 0x7f50140ae490 2024-03-29T11:50:48.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f501a7eb640 1 librados: init done 2024-03-29T11:50:48.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.692+0000 7f50137fe640 4 set_mon_vals no callback set 2024-03-29T11:50:48.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.693+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f5008004a40 con 0x7f50140ae490 2024-03-29T11:50:48.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.693+0000 7f5011ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.693+0000 7f5011ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.693+0000 7f5011ffb640 10 monclient: dump: 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.694+0000 7f501a7eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.694+0000 7f501a7eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.694+0000 7f501a7eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.694+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50140ae870 con 0x7f50140ae490 2024-03-29T11:50:48.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.694+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f500801aad0 con 0x7f50140ae490 2024-03-29T11:50:48.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.695+0000 7f50197e9640 10 monclient: get_auth_request con 0x7f500006dca0 auth_method 0 2024-03-29T11:50:48.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.695+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f5008084a50 con 0x7f50140ae490 2024-03-29T11:50:48.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.695+0000 7f5011ffb640 10 client.4522.objecter ms_dispatch 0x7f5014000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:48.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.696+0000 7f5011ffb640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:48.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.696+0000 7f5011ffb640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:48.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.696+0000 7f5011ffb640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-29T11:50:48.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.696+0000 7f5011ffb640 10 client.4522.objecter ms_handle_connect 0x7f500006dca0 2024-03-29T11:50:48.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.704+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5008050160 con 0x7f50140ae490 2024-03-29T11:50:48.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.704+0000 7f5011ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:48.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:48.704+0000 7f5011ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:49.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.097+0000 7f501a7eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:49.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.097+0000 7f501a7eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:49.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.097+0000 7f501a7eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:50:49.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.097+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f501406f5a0 con 0x7f50140ae490 2024-03-29T11:50:49.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.097+0000 7f5011ffb640 1 -- 172.21.15.55:0/2687891806 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f5008052260 con 0x7f50140ae490 2024-03-29T11:50:49.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.097+0000 7f5011ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:49.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.097+0000 7f5011ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:49.106 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:50:49.106 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:49.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.104+0000 7f501a7eb640 10 librados: watch_flush enter 2024-03-29T11:50:49.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.107+0000 7f501a7eb640 10 librados: watch_flush exit 2024-03-29T11:50:49.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.107+0000 7f501a7eb640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-29T11:50:49.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.107+0000 7f501a7eb640 10 client.4522.objecter successfully canceled tick 2024-03-29T11:50:49.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.107+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 >> v1:172.21.15.171:6816/1463197163 conn(0x7f500006dca0 legacy=0x7f5000070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:49.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.107+0000 7f501a7eb640 10 monclient: shutdown 2024-03-29T11:50:49.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.107+0000 7f501a7eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:49.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.107+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 >> v1:172.21.15.171:6789/0 conn(0x7f50140ae490 legacy=0x7f501406ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.108+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 shutdown_connections 2024-03-29T11:50:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.108+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 >> 172.21.15.55:0/2687891806 conn(0x7f50140673e0 msgr2=0x7f50140b1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.109+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 shutdown_connections 2024-03-29T11:50:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.109+0000 7f501a7eb640 1 -- 172.21.15.55:0/2687891806 wait complete. 2024-03-29T11:50:49.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:49.109+0000 7f501a7eb640 1 librados: shutdown 2024-03-29T11:50:49.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:55.942 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:56.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.088+0000 7f71c46ff640 10 monclient: get_monmap_and_config 2024-03-29T11:50:56.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.088+0000 7f71c46ff640 10 monclient: build_initial_monmap 2024-03-29T11:50:56.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.088+0000 7f71c46ff640 10 monclient: monmap: 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:50:56.089972+0000 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:50:56.089972+0000 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:50:56.096 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:56.097 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.089+0000 7f71c46ff640 1 Processor -- start 2024-03-29T11:50:56.097 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.089+0000 7f71c46ff640 1 -- start start 2024-03-29T11:50:56.097 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.089+0000 7f71c46ff640 10 monclient: init 2024-03-29T11:50:56.097 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 10 monclient(hunting): picked mon.noname-b con 0x7f71bc1422b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 10 monclient(hunting): picked mon.noname-a con 0x7f71bc142690 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 10 monclient(hunting): picked mon.noname-c con 0x7f71bc0b17d0 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc142c80 con 0x7f71bc142690 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc0aae40 con 0x7f71bc0b17d0 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc13fbd0 con 0x7f71bc1422b0 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.090+0000 7f71c46ff640 10 monclient(hunting): authenticate will time out at 1469.370624s 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c249b640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f71bc1422b0 0x7f71bc13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:40536/0 (socket says 172.21.15.55:40536) 2024-03-29T11:50:56.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c249b640 1 -- 172.21.15.55:0/3759140618 learned_addr learned my addr 172.21.15.55:0/3759140618 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:56.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810710096 0 0) 0x7f71bc142c80 con 0x7f71bc142690 2024-03-29T11:50:56.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient(hunting): _negotiate 2024-03-29T11:50:56.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:56.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:56.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:50:56.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a8003520 con 0x7f71bc142690 2024-03-29T11:50:56.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82224507 0 0) 0x7f71bc13fbd0 con 0x7f71bc1422b0 2024-03-29T11:50:56.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient(hunting): _negotiate 2024-03-29T11:50:56.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:56.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:56.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:50:56.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc142c80 con 0x7f71bc1422b0 2024-03-29T11:50:56.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1686261667 0 0) 0x7f71a8003520 con 0x7f71bc142690 2024-03-29T11:50:56.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 >> v1:172.21.15.55:6790/0 conn(0x7f71bc1422b0 legacy=0x7f71bc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:56.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 >> v1:172.21.15.171:6789/0 conn(0x7f71bc0b17d0 legacy=0x7f71bc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:50:56.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient: _finish_hunting 0 2024-03-29T11:50:56.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 monclient: found mon.noname-a 2024-03-29T11:50:56.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:56.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:50:56.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc151940 con 0x7f71bc142690 2024-03-29T11:50:56.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient: _finish_auth 0 2024-03-29T11:50:56.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient: _check_auth_tickets 2024-03-29T11:50:56.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:56.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f71ac001690 con 0x7f71bc142690 2024-03-29T11:50:56.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:56.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:50:56.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c1499640 10 monclient: dump: 2024-03-29T11:50:56.104 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:56.104 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:56.104 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.104 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.104 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:56.104 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:56.105 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:56.105 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:56.105 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:56.105 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:56.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c46ff640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:50:56.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.091+0000 7f71c46ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:56.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.092+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71ac0017f0 con 0x7f71bc142690 2024-03-29T11:50:56.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.092+0000 7f71c1499640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:56.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.092+0000 7f71c1499640 1 -- 172.21.15.55:0/3759140618 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f71ac001b80 con 0x7f71bc142690 2024-03-29T11:50:56.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.092+0000 7f71c1499640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:56.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.092+0000 7f71c1499640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:56.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.092+0000 7f71c1499640 10 monclient: dump: 2024-03-29T11:50:56.106 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:56.106 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:56.106 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.106 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.106 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:56.106 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.092+0000 7f71c46ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.092+0000 7f71c46ff640 4 set_mon_vals no callback set 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.093+0000 7f71c46ff640 10 monclient: shutdown 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.093+0000 7f71c46ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.093+0000 7f71c46ff640 1 -- 172.21.15.55:0/3759140618 >> v1:172.21.15.55:6789/0 conn(0x7f71bc142690 legacy=0x7f71bc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:56.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.093+0000 7f71c46ff640 1 -- 172.21.15.55:0/3759140618 shutdown_connections 2024-03-29T11:50:56.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.093+0000 7f71c46ff640 1 -- 172.21.15.55:0/3759140618 >> 172.21.15.55:0/3759140618 conn(0x7f71bc066a00 msgr2=0x7f71bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:56.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.093+0000 7f71c46ff640 1 -- 172.21.15.55:0/3759140618 shutdown_connections 2024-03-29T11:50:56.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.094+0000 7f71c46ff640 1 -- 172.21.15.55:0/3759140618 wait complete. 2024-03-29T11:50:56.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.095+0000 7f71c46ff640 10 monclient: build_initial_monmap 2024-03-29T11:50:56.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.095+0000 7f71c46ff640 10 monclient: monmap: 2024-03-29T11:50:56.113 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:50:56.113 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:56.113 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:50:56.113 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:50:56.113 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:56.113 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.095+0000 7f71c46ff640 1 librados: starting msgr at 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.095+0000 7f71c46ff640 1 librados: starting objecter 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.095+0000 7f71c46ff640 1 Processor -- start 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.095+0000 7f71c46ff640 1 -- start start 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.095+0000 7f71c46ff640 1 librados: setting wanted keys 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.095+0000 7f71c46ff640 1 librados: calling monclient init 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.095+0000 7f71c46ff640 10 monclient: init 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.096+0000 7f71c46ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.096+0000 7f71c46ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:50:56.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.096+0000 7f71c46ff640 10 monclient(hunting): picked mon.noname-c con 0x7f71bc0b17d0 addr v1:172.21.15.55:6790/0 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.096+0000 7f71c46ff640 10 monclient(hunting): picked mon.noname-b con 0x7f71bc1422b0 addr v1:172.21.15.171:6789/0 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71c46ff640 10 monclient(hunting): picked mon.noname-a con 0x7f71bc142690 addr v1:172.21.15.55:6789/0 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71c46ff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc132810 con 0x7f71bc142690 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71c2c9c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f71bc142690 0x7f71bc1ddd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:55824/0 (socket says 172.21.15.55:55824) 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71c2c9c640 1 -- 172.21.15.55:0/786760414 learned_addr learned my addr 172.21.15.55:0/786760414 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc132af0 con 0x7f71bc1422b0 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc06e0e0 con 0x7f71bc0b17d0 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71c46ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71c46ff640 10 monclient(hunting): authenticate will time out at 1469.376909s 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 client.?.objecter ms_handle_connect 0x7f71bc142690 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 client.?.objecter ms_handle_connect 0x7f71bc0b17d0 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979598501 0 0) 0x7f71bc132810 con 0x7f71bc142690 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:56.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a0004eb0 con 0x7f71bc142690 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208889046 0 0) 0x7f71bc06e0e0 con 0x7f71bc0b17d0 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.097+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc132810 con 0x7f71bc0b17d0 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1916678259 0 0) 0x7f71bc132810 con 0x7f71bc0b17d0 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71bc06e0e0 con 0x7f71bc0b17d0 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f71b8001930 con 0x7f71bc0b17d0 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:56.116 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3073199968 0 0) 0x7f71a0004eb0 con 0x7f71bc142690 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71bc132810 con 0x7f71bc142690 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f71b4001c60 con 0x7f71bc142690 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:56.117 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2511774810 0 0) 0x7f71bc06e0e0 con 0x7f71bc0b17d0 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 >> v1:172.21.15.171:6789/0 conn(0x7f71bc1422b0 legacy=0x7f71bc1dd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 >> v1:172.21.15.55:6789/0 conn(0x7f71bc142690 legacy=0x7f71bc1ddd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 monclient: found mon.c 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:56.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc1dd100 con 0x7f71bc0b17d0 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71c46ff640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71c46ff640 10 monclient: _renew_subs 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71c46ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc132470 con 0x7f71bc0b17d0 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71c46ff640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71c46ff640 10 monclient: _renew_subs 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71c46ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc13bc10 con 0x7f71bc0b17d0 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.098+0000 7f71c46ff640 1 librados: init done 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.099+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71b8001a90 con 0x7f71bc0b17d0 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.099+0000 7f71b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.099+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f71b8004980 con 0x7f71bc0b17d0 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.099+0000 7f71b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.099+0000 7f71b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.099+0000 7f71b2ffd640 10 monclient: dump: 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.099+0000 7f71b3fff640 4 set_mon_vals no callback set 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.100+0000 7f71c46ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.101+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f71b801a950 con 0x7f71bc0b17d0 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.101+0000 7f71c46ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.101+0000 7f71c46ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.101+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a4005080 con 0x7f71bc0b17d0 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.102+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f71b8084a20 con 0x7f71bc0b17d0 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.102+0000 7f71b2ffd640 10 client.4559.objecter ms_dispatch 0x7f71bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.102+0000 7f71b2ffd640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.102+0000 7f71b2ffd640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.102+0000 7f71b2ffd640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.102+0000 7f71c1c9a640 10 monclient: get_auth_request con 0x7f71a006de30 auth_method 0 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.102+0000 7f71b2ffd640 10 client.4559.objecter ms_handle_connect 0x7f71a006de30 2024-03-29T11:50:56.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.111+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71b8050300 con 0x7f71bc0b17d0 2024-03-29T11:50:56.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.111+0000 7f71b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:56.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.111+0000 7f71b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:56.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.473+0000 7f71c46ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:56.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.474+0000 7f71c46ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:56.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.474+0000 7f71c46ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:50:56.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.474+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71a4005330 con 0x7f71bc0b17d0 2024-03-29T11:50:56.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.475+0000 7f71b2ffd640 1 -- 172.21.15.55:0/786760414 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f71b8052230 con 0x7f71bc0b17d0 2024-03-29T11:50:56.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.475+0000 7f71b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:56.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.475+0000 7f71b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:56.482 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:50:56.482 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:56.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 10 librados: watch_flush enter 2024-03-29T11:50:56.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 10 librados: watch_flush exit 2024-03-29T11:50:56.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:50:56.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:50:56.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 >> v1:172.21.15.171:6816/1463197163 conn(0x7f71a006de30 legacy=0x7f71a00702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:56.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 10 monclient: shutdown 2024-03-29T11:50:56.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:56.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 >> v1:172.21.15.55:6790/0 conn(0x7f71bc0b17d0 legacy=0x7f71bc12dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:56.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 shutdown_connections 2024-03-29T11:50:56.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.478+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 >> 172.21.15.55:0/786760414 conn(0x7f71bc066a00 msgr2=0x7f71bc0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:56.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.479+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 shutdown_connections 2024-03-29T11:50:56.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.479+0000 7f71c46ff640 1 -- 172.21.15.55:0/786760414 wait complete. 2024-03-29T11:50:56.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:50:56.479+0000 7f71c46ff640 1 librados: shutdown 2024-03-29T11:50:56.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:03.309 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.459+0000 7f2e6d8f4640 10 monclient: get_monmap_and_config 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.459+0000 7f2e6d8f4640 10 monclient: build_initial_monmap 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.459+0000 7f2e6d8f4640 10 monclient: monmap: 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:51:03.460257+0000 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:51:03.460257+0000 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:51:03.466 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:03.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.459+0000 7f2e6d8f4640 1 Processor -- start 2024-03-29T11:51:03.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.460+0000 7f2e6d8f4640 1 -- start start 2024-03-29T11:51:03.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.460+0000 7f2e6d8f4640 10 monclient: init 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 10 monclient(hunting): picked mon.noname-c con 0x7f2e680b1f60 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 10 monclient(hunting): picked mon.noname-b con 0x7f2e68071eb0 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 10 monclient(hunting): picked mon.noname-a con 0x7f2e68143d10 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6806ef80 con 0x7f2e68143d10 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6806f260 con 0x7f2e680b1f60 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e680b40c0 con 0x7f2e68071eb0 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e6d8f4640 10 monclient(hunting): authenticate will time out at 1476.740801s 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e677fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2e68143d10 0x7f2e68146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41852/0 (socket says 172.21.15.55:41852) 2024-03-29T11:51:03.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e677fe640 1 -- 172.21.15.55:0/1749201457 learned_addr learned my addr 172.21.15.55:0/1749201457 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:03.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165639175 0 0) 0x7f2e6806f260 con 0x7f2e680b1f60 2024-03-29T11:51:03.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:51:03.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:03.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:03.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e65ffb640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:51:03.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.461+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e54003520 con 0x7f2e680b1f60 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886854259 0 0) 0x7f2e680b40c0 con 0x7f2e68071eb0 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6806f260 con 0x7f2e68071eb0 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519430955 0 0) 0x7f2e54003520 con 0x7f2e680b1f60 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 >> v1:172.21.15.55:6790/0 conn(0x7f2e68071eb0 legacy=0x7f2e68146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 >> v1:172.21.15.55:6789/0 conn(0x7f2e68143d10 legacy=0x7f2e68146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 monclient: found mon.noname-c 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68155d70 con 0x7f2e680b1f60 2024-03-29T11:51:03.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: _finish_auth 0 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e6d8f4640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e6d8f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2e60001690 con 0x7f2e680b1f60 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: dump: 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:03.472 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e6d8f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e600018b0 con 0x7f2e680b1f60 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e6d8f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e6d8f4640 4 set_mon_vals no callback set 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 1 -- 172.21.15.55:0/1749201457 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2e60001c40 con 0x7f2e680b1f60 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.462+0000 7f2e65ffb640 10 monclient: dump: 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.473 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.463+0000 7f2e6d8f4640 10 monclient: shutdown 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.463+0000 7f2e6d8f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.463+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1749201457 >> v1:172.21.15.171:6789/0 conn(0x7f2e680b1f60 legacy=0x7f2e68143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.463+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1749201457 shutdown_connections 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.463+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1749201457 >> 172.21.15.55:0/1749201457 conn(0x7f2e68066a00 msgr2=0x7f2e68066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:03.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.463+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1749201457 shutdown_connections 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.463+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1749201457 wait complete. 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.464+0000 7f2e6d8f4640 10 monclient: build_initial_monmap 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.464+0000 7f2e6d8f4640 10 monclient: monmap: 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:03.478 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.465+0000 7f2e6d8f4640 1 librados: starting msgr at 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.465+0000 7f2e6d8f4640 1 librados: starting objecter 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.465+0000 7f2e6d8f4640 1 Processor -- start 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.465+0000 7f2e6d8f4640 1 -- start start 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.465+0000 7f2e6d8f4640 1 librados: setting wanted keys 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.465+0000 7f2e6d8f4640 1 librados: calling monclient init 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.465+0000 7f2e6d8f4640 10 monclient: init 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 10 monclient(hunting): picked mon.noname-b con 0x7f2e680b1f60 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 10 monclient(hunting): picked mon.noname-c con 0x7f2e68143d10 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:03.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 10 monclient(hunting): picked mon.noname-a con 0x7f2e681df440 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e68149ff0 con 0x7f2e681df440 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6813bb90 con 0x7f2e680b1f60 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6814b6b0 con 0x7f2e68143d10 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e6d8f4640 10 monclient(hunting): authenticate will time out at 1476.746061s 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e66ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2e680b1f60 0x7f2e681ded30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:55574/0 (socket says 172.21.15.55:55574) 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e66ffd640 1 -- 172.21.15.55:0/1980048752 learned_addr learned my addr 172.21.15.55:0/1980048752 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e5ffff640 10 client.?.objecter ms_handle_connect 0x7f2e681df440 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e5ffff640 10 client.?.objecter ms_handle_connect 0x7f2e680b1f60 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e5ffff640 10 client.?.objecter ms_handle_connect 0x7f2e68143d10 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.466+0000 7f2e5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633208321 0 0) 0x7f2e6814b6b0 con 0x7f2e68143d10 2024-03-29T11:51:03.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e48004eb0 con 0x7f2e68143d10 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589069277 0 0) 0x7f2e68149ff0 con 0x7f2e681df440 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6814b6b0 con 0x7f2e681df440 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3891791950 0 0) 0x7f2e48004eb0 con 0x7f2e68143d10 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e68149ff0 con 0x7f2e68143d10 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2e58001cb0 con 0x7f2e68143d10 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:03.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): dump: 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859127484 0 0) 0x7f2e6813bb90 con 0x7f2e680b1f60 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:03.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.467+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e48004eb0 con 0x7f2e680b1f60 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1896703830 0 0) 0x7f2e6814b6b0 con 0x7f2e681df440 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e6813bb90 con 0x7f2e681df440 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2e50001780 con 0x7f2e681df440 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 10 monclient(hunting): dump: 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:03.483 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 345173691 0 0) 0x7f2e6813bb90 con 0x7f2e681df440 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 >> v1:172.21.15.55:6790/0 conn(0x7f2e68143d10 legacy=0x7f2e6813ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 >> v1:172.21.15.171:6789/0 conn(0x7f2e680b1f60 legacy=0x7f2e681ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 1 monclient: found mon.a 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68149b80 con 0x7f2e681df440 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 10 monclient: _finish_auth 0 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e6d8f4640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e6d8f4640 10 monclient: _renew_subs 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e6d8f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:03.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6814a480 con 0x7f2e681df440 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e6d8f4640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e6d8f4640 10 monclient: _renew_subs 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e6d8f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e68154f40 con 0x7f2e681df440 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.468+0000 7f2e6d8f4640 1 librados: init done 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.469+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e50001b90 con 0x7f2e681df440 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e64ff9640 4 set_mon_vals no callback set 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f2e50004980 con 0x7f2e681df440 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e5ffff640 10 monclient: dump: 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:03.485 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e6d8f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e6d8f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e6d8f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:03.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.470+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c005080 con 0x7f2e681df440 2024-03-29T11:51:03.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.472+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f2e50004ae0 con 0x7f2e681df440 2024-03-29T11:51:03.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.473+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f2e50084800 con 0x7f2e681df440 2024-03-29T11:51:03.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.473+0000 7f2e5ffff640 10 client.4560.objecter ms_dispatch 0x7f2e68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:03.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.473+0000 7f2e5ffff640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:03.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.473+0000 7f2e5ffff640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:03.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.473+0000 7f2e66ffd640 10 monclient: get_auth_request con 0x7f2e4806ddd0 auth_method 0 2024-03-29T11:51:03.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.473+0000 7f2e5ffff640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T11:51:03.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.473+0000 7f2e5ffff640 10 client.4560.objecter ms_handle_connect 0x7f2e4806ddd0 2024-03-29T11:51:03.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.479+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e500500e0 con 0x7f2e681df440 2024-03-29T11:51:03.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.479+0000 7f2e5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:03.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.479+0000 7f2e5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:03.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.845+0000 7f2e6d8f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:03.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.845+0000 7f2e6d8f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:03.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.845+0000 7f2e6d8f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:03.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.845+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e4c005330 con 0x7f2e681df440 2024-03-29T11:51:03.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.846+0000 7f2e5ffff640 1 -- 172.21.15.55:0/1980048752 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f2e50052010 con 0x7f2e681df440 2024-03-29T11:51:03.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.846+0000 7f2e5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:03.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.846+0000 7f2e5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:03.853 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:51:03.853 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:03.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.849+0000 7f2e6d8f4640 10 librados: watch_flush enter 2024-03-29T11:51:03.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.849+0000 7f2e6d8f4640 10 librados: watch_flush exit 2024-03-29T11:51:03.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.849+0000 7f2e6d8f4640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T11:51:03.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 10 client.4560.objecter successfully canceled tick 2024-03-29T11:51:03.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 >> v1:172.21.15.171:6816/1463197163 conn(0x7f2e4806ddd0 legacy=0x7f2e48070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:03.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 10 monclient: shutdown 2024-03-29T11:51:03.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:03.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 >> v1:172.21.15.55:6789/0 conn(0x7f2e681df440 legacy=0x7f2e6813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:03.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 shutdown_connections 2024-03-29T11:51:03.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 >> 172.21.15.55:0/1980048752 conn(0x7f2e68066a00 msgr2=0x7f2e680b7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:03.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 shutdown_connections 2024-03-29T11:51:03.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 1 -- 172.21.15.55:0/1980048752 wait complete. 2024-03-29T11:51:03.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:03.850+0000 7f2e6d8f4640 1 librados: shutdown 2024-03-29T11:51:03.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:10.676 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.824+0000 7f92af76c640 10 monclient: get_monmap_and_config 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.824+0000 7f92af76c640 10 monclient: build_initial_monmap 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.824+0000 7f92af76c640 10 monclient: monmap: 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:51:10.825954+0000 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:51:10.825954+0000 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:51:10.832 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:10.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.826+0000 7f92af76c640 1 Processor -- start 2024-03-29T11:51:10.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.827+0000 7f92af76c640 1 -- start start 2024-03-29T11:51:10.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.827+0000 7f92af76c640 10 monclient: init 2024-03-29T11:51:10.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:10.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:51:10.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 10 monclient(hunting): picked mon.noname-a con 0x7f92a80b0500 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:10.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 10 monclient(hunting): picked mon.noname-c con 0x7f92a80b08e0 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:10.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 10 monclient(hunting): picked mon.noname-b con 0x7f92a80aeca0 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:10.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a8140620 con 0x7f92a80b0500 2024-03-29T11:51:10.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a813daf0 con 0x7f92a80b08e0 2024-03-29T11:51:10.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a80ae180 con 0x7f92a80aeca0 2024-03-29T11:51:10.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:10.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.828+0000 7f92af76c640 10 monclient(hunting): authenticate will time out at 1484.108438s 2024-03-29T11:51:10.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.833+0000 7f92add09640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f92a80aeca0 0x7f92a814b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:44397/0 (socket says 172.21.15.55:44397) 2024-03-29T11:51:10.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.833+0000 7f92add09640 1 -- 172.21.15.55:0/1937115302 learned_addr learned my addr 172.21.15.55:0/1937115302 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:10.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.835+0000 7f929ffff640 1 -- 172.21.15.55:0/1937115302 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429875383 0 0) 0x7f92a80ae180 con 0x7f92a80aeca0 2024-03-29T11:51:10.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.835+0000 7f929ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:51:10.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.835+0000 7f929ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:10.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.835+0000 7f929ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:10.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.835+0000 7f929ffff640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:51:10.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.835+0000 7f929ffff640 1 -- 172.21.15.55:0/1937115302 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9294003520 con 0x7f92a80aeca0 2024-03-29T11:51:10.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 1 -- 172.21.15.55:0/1937115302 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2407170038 0 0) 0x7f9294003520 con 0x7f92a80aeca0 2024-03-29T11:51:10.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 1 -- 172.21.15.55:0/1937115302 >> v1:172.21.15.171:6789/0 conn(0x7f92a80b08e0 legacy=0x7f92a813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:10.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 1 -- 172.21.15.55:0/1937115302 >> v1:172.21.15.55:6789/0 conn(0x7f92a80b0500 legacy=0x7f92a813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:10.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:51:10.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 1 monclient: found mon.noname-b 2024-03-29T11:51:10.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 1 -- 172.21.15.55:0/1937115302 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a8072510 con 0x7f92a80aeca0 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: _finish_auth 0 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 1 -- 172.21.15.55:0/1937115302 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92a0001a10 con 0x7f92a80aeca0 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: dump: 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:10.844 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f92af76c640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f92af76c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 1 -- 172.21.15.55:0/1937115302 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a0001be0 con 0x7f92a80aeca0 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 1 -- 172.21.15.55:0/1937115302 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92a0001ff0 con 0x7f92a80aeca0 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f929ffff640 10 monclient: dump: 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:10.845 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f92af76c640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:10.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.836+0000 7f92af76c640 4 set_mon_vals no callback set 2024-03-29T11:51:10.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.837+0000 7f92af76c640 10 monclient: shutdown 2024-03-29T11:51:10.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.837+0000 7f92af76c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:10.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.837+0000 7f92af76c640 1 -- 172.21.15.55:0/1937115302 >> v1:172.21.15.55:6790/0 conn(0x7f92a80aeca0 legacy=0x7f92a814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:10.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.837+0000 7f92af76c640 1 -- 172.21.15.55:0/1937115302 shutdown_connections 2024-03-29T11:51:10.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.837+0000 7f92af76c640 1 -- 172.21.15.55:0/1937115302 >> 172.21.15.55:0/1937115302 conn(0x7f92a806b660 msgr2=0x7f92a806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:10.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.839+0000 7f92af76c640 1 -- 172.21.15.55:0/1937115302 shutdown_connections 2024-03-29T11:51:10.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.839+0000 7f92af76c640 1 -- 172.21.15.55:0/1937115302 wait complete. 2024-03-29T11:51:10.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.840+0000 7f92af76c640 10 monclient: build_initial_monmap 2024-03-29T11:51:10.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.840+0000 7f92af76c640 10 monclient: monmap: 2024-03-29T11:51:10.848 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:10.848 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:10.848 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:51:10.848 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:51:10.848 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:10.848 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:10.848 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:10.849 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:51:10.849 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:51:10.849 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:10.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.840+0000 7f92af76c640 1 librados: starting msgr at 2024-03-29T11:51:10.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.840+0000 7f92af76c640 1 librados: starting objecter 2024-03-29T11:51:10.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.840+0000 7f92af76c640 1 Processor -- start 2024-03-29T11:51:10.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.840+0000 7f92af76c640 1 -- start start 2024-03-29T11:51:10.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.840+0000 7f92af76c640 1 librados: setting wanted keys 2024-03-29T11:51:10.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.840+0000 7f92af76c640 1 librados: calling monclient init 2024-03-29T11:51:10.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.840+0000 7f92af76c640 10 monclient: init 2024-03-29T11:51:10.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:10.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:51:10.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 10 monclient(hunting): picked mon.noname-b con 0x7f92a80aeca0 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:10.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 10 monclient(hunting): picked mon.noname-c con 0x7f92a80b0500 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:10.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 10 monclient(hunting): picked mon.noname-a con 0x7f92a81e0090 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:10.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a8145580 con 0x7f92a81e0090 2024-03-29T11:51:10.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a8144c00 con 0x7f92a80aeca0 2024-03-29T11:51:10.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a8143b30 con 0x7f92a80b0500 2024-03-29T11:51:10.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:10.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.841+0000 7f92af76c640 10 monclient(hunting): authenticate will time out at 1484.121360s 2024-03-29T11:51:10.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.845+0000 7f92ad508640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f92a80aeca0 0x7f92a8143de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:35411/0 (socket says 172.21.15.55:35411) 2024-03-29T11:51:10.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.846+0000 7f92ad508640 1 -- 172.21.15.55:0/3581374601 learned_addr learned my addr 172.21.15.55:0/3581374601 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:10.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.846+0000 7f929dffb640 10 client.?.objecter ms_handle_connect 0x7f92a80aeca0 2024-03-29T11:51:10.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.846+0000 7f929dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:10.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702970334 0 0) 0x7f92a8144c00 con 0x7f92a80aeca0 2024-03-29T11:51:10.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:51:10.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:10.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:10.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:51:10.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f928c004eb0 con 0x7f92a80aeca0 2024-03-29T11:51:10.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596251995 0 0) 0x7f928c004eb0 con 0x7f92a80aeca0 2024-03-29T11:51:10.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f928c005300 con 0x7f92a80aeca0 2024-03-29T11:51:10.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92a4004040 con 0x7f92a80aeca0 2024-03-29T11:51:10.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 10 monclient(hunting): dump: 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.847+0000 7f929dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.848+0000 7f929dffb640 10 client.?.objecter ms_handle_connect 0x7f92a80b0500 2024-03-29T11:51:10.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.848+0000 7f929dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.848+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2878253236 0 0) 0x7f928c005300 con 0x7f92a80aeca0 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.848+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 >> v1:172.21.15.55:6790/0 conn(0x7f92a80b0500 legacy=0x7f92a81444f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.848+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 >> v1:172.21.15.55:6789/0 conn(0x7f92a81e0090 legacy=0x7f92a81e2480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f929dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f929dffb640 1 monclient: found mon.b 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f929dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f929dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a81457f0 con 0x7f92a80aeca0 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f929dffb640 10 monclient: _finish_auth 0 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f929dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f929dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f92af76c640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f92af76c640 10 monclient: _renew_subs 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f92af76c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:10.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f92af76c640 1 -- 172.21.15.55:0/3581374601 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a8145ad0 con 0x7f92a80aeca0 2024-03-29T11:51:10.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f92af76c640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:10.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f92af76c640 10 monclient: _renew_subs 2024-03-29T11:51:10.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f92af76c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:10.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f92af76c640 1 -- 172.21.15.55:0/3581374601 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a814bfc0 con 0x7f92a80aeca0 2024-03-29T11:51:10.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.850+0000 7f92af76c640 1 librados: init done 2024-03-29T11:51:10.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a4000ed0 con 0x7f92a80aeca0 2024-03-29T11:51:10.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f929dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:10.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f92a4004040 con 0x7f92a80aeca0 2024-03-29T11:51:10.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f929dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:10.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f929dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:10.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f929dffb640 10 monclient: dump: 2024-03-29T11:51:10.862 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f92937fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f92937fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f92937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f92937fe640 1 -- 172.21.15.55:0/3581374601 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a80ad0c0 con 0x7f92a80aeca0 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.852+0000 7f929f7fe640 4 set_mon_vals no callback set 2024-03-29T11:51:10.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.855+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f92a4001140 con 0x7f92a80aeca0 2024-03-29T11:51:10.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.856+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f92a4084770 con 0x7f92a80aeca0 2024-03-29T11:51:10.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.856+0000 7f929dffb640 10 client.4534.objecter ms_dispatch 0x7f92a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:10.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.856+0000 7f929dffb640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:10.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.856+0000 7f929dffb640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:10.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.856+0000 7f929dffb640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T11:51:10.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.859+0000 7f92acd07640 10 monclient: get_auth_request con 0x7f928c06e0e0 auth_method 0 2024-03-29T11:51:10.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.860+0000 7f929dffb640 10 client.4534.objecter ms_handle_connect 0x7f928c06e0e0 2024-03-29T11:51:10.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.863+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92a404fe80 con 0x7f92a80aeca0 2024-03-29T11:51:10.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.863+0000 7f929dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:10.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:10.863+0000 7f929dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:11.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.194+0000 7f92937fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:11.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.194+0000 7f92937fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:11.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.194+0000 7f92937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:11.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.194+0000 7f92937fe640 1 -- 172.21.15.55:0/3581374601 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92a8075f10 con 0x7f92a80aeca0 2024-03-29T11:51:11.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.196+0000 7f929dffb640 1 -- 172.21.15.55:0/3581374601 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f92a4057ba0 con 0x7f92a80aeca0 2024-03-29T11:51:11.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.196+0000 7f929dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:11.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.196+0000 7f929dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:11.203 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:51:11.204 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:11.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.200+0000 7f92937fe640 10 librados: watch_flush enter 2024-03-29T11:51:11.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.200+0000 7f92937fe640 10 librados: watch_flush exit 2024-03-29T11:51:11.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.200+0000 7f92937fe640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T11:51:11.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.200+0000 7f92937fe640 10 client.4534.objecter successfully canceled tick 2024-03-29T11:51:11.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.201+0000 7f92937fe640 1 -- 172.21.15.55:0/3581374601 >> v1:172.21.15.171:6816/1463197163 conn(0x7f928c06e0e0 legacy=0x7f928c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:11.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.201+0000 7f92937fe640 10 monclient: shutdown 2024-03-29T11:51:11.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.201+0000 7f92937fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:11.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.201+0000 7f92937fe640 1 -- 172.21.15.55:0/3581374601 >> v1:172.21.15.171:6789/0 conn(0x7f92a80aeca0 legacy=0x7f92a8143de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:11.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.202+0000 7f92937fe640 1 -- 172.21.15.55:0/3581374601 shutdown_connections 2024-03-29T11:51:11.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.202+0000 7f92937fe640 1 -- 172.21.15.55:0/3581374601 >> 172.21.15.55:0/3581374601 conn(0x7f92a806b660 msgr2=0x7f92a80b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:11.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.202+0000 7f92937fe640 1 -- 172.21.15.55:0/3581374601 shutdown_connections 2024-03-29T11:51:11.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.202+0000 7f92937fe640 1 -- 172.21.15.55:0/3581374601 wait complete. 2024-03-29T11:51:11.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:11.204+0000 7f92937fe640 1 librados: shutdown 2024-03-29T11:51:11.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:18.035 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:18.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.174+0000 7f3c23fff640 10 monclient: get_monmap_and_config 2024-03-29T11:51:18.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.174+0000 7f3c23fff640 10 monclient: build_initial_monmap 2024-03-29T11:51:18.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.174+0000 7f3c23fff640 10 monclient: monmap: 2024-03-29T11:51:18.180 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:18.180 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:18.180 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:51:18.174515+0000 2024-03-29T11:51:18.181 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:51:18.174515+0000 2024-03-29T11:51:18.181 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:18.181 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:18.181 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:18.181 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:51:18.181 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:51:18.181 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:18.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.177+0000 7f3c23fff640 1 Processor -- start 2024-03-29T11:51:18.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.177+0000 7f3c23fff640 1 -- start start 2024-03-29T11:51:18.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.177+0000 7f3c23fff640 10 monclient: init 2024-03-29T11:51:18.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:18.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:51:18.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 10 monclient(hunting): picked mon.noname-b con 0x7f3c240b2800 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:18.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 10 monclient(hunting): picked mon.noname-c con 0x7f3c240a9c00 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:18.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 10 monclient(hunting): picked mon.noname-a con 0x7f3c240b38c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:18.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c24141eb0 con 0x7f3c240b38c0 2024-03-29T11:51:18.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c240b22d0 con 0x7f3c240a9c00 2024-03-29T11:51:18.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c24142d30 con 0x7f3c240b2800 2024-03-29T11:51:18.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:18.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.178+0000 7f3c23fff640 10 monclient(hunting): authenticate will time out at 1491.456924s 2024-03-29T11:51:18.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.179+0000 7f3c22ffd640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f3c240b2800 0x7f3c24142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:39250/0 (socket says 172.21.15.55:39250) 2024-03-29T11:51:18.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.179+0000 7f3c22ffd640 1 -- 172.21.15.55:0/1370427040 learned_addr learned my addr 172.21.15.55:0/1370427040 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:18.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.180+0000 7f3c21ffb640 1 -- 172.21.15.55:0/1370427040 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970762060 0 0) 0x7f3c24142d30 con 0x7f3c240b2800 2024-03-29T11:51:18.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.181+0000 7f3c21ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:51:18.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.181+0000 7f3c21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:18.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.181+0000 7f3c21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:18.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.181+0000 7f3c21ffb640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:51:18.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.181+0000 7f3c21ffb640 1 -- 172.21.15.55:0/1370427040 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c0c003520 con 0x7f3c240b2800 2024-03-29T11:51:18.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.182+0000 7f3c21ffb640 1 -- 172.21.15.55:0/1370427040 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2331135213 0 0) 0x7f3c0c003520 con 0x7f3c240b2800 2024-03-29T11:51:18.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.182+0000 7f3c21ffb640 1 -- 172.21.15.55:0/1370427040 >> v1:172.21.15.171:6789/0 conn(0x7f3c240a9c00 legacy=0x7f3c2413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:18.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.182+0000 7f3c21ffb640 1 -- 172.21.15.55:0/1370427040 >> v1:172.21.15.55:6789/0 conn(0x7f3c240b38c0 legacy=0x7f3c2414d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:51:18.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:51:18.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 1 monclient: found mon.noname-b 2024-03-29T11:51:18.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 1 -- 172.21.15.55:0/1370427040 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c240aaf90 con 0x7f3c240b2800 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 10 monclient: _finish_auth 0 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c23fff640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c23fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 1 -- 172.21.15.55:0/1370427040 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3c18001690 con 0x7f3c240b2800 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c21ffb640 10 monclient: dump: 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:18.191 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.183+0000 7f3c23fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c21ffb640 1 -- 172.21.15.55:0/1370427040 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c180017f0 con 0x7f3c240b2800 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c23fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:18.192 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c23fff640 4 set_mon_vals no callback set 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c21ffb640 1 -- 172.21.15.55:0/1370427040 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3c18001c40 con 0x7f3c240b2800 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c21ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c21ffb640 10 monclient: dump: 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:18.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c23fff640 10 monclient: shutdown 2024-03-29T11:51:18.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c23fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:18.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.184+0000 7f3c23fff640 1 -- 172.21.15.55:0/1370427040 >> v1:172.21.15.55:6790/0 conn(0x7f3c240b2800 legacy=0x7f3c24142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:18.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.185+0000 7f3c23fff640 1 -- 172.21.15.55:0/1370427040 shutdown_connections 2024-03-29T11:51:18.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.185+0000 7f3c23fff640 1 -- 172.21.15.55:0/1370427040 >> 172.21.15.55:0/1370427040 conn(0x7f3c24066a00 msgr2=0x7f3c24066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:18.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.185+0000 7f3c23fff640 1 -- 172.21.15.55:0/1370427040 shutdown_connections 2024-03-29T11:51:18.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.185+0000 7f3c23fff640 1 -- 172.21.15.55:0/1370427040 wait complete. 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.186+0000 7f3c23fff640 10 monclient: build_initial_monmap 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.186+0000 7f3c23fff640 10 monclient: monmap: 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.186+0000 7f3c23fff640 1 librados: starting msgr at 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.186+0000 7f3c23fff640 1 librados: starting objecter 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.186+0000 7f3c23fff640 1 Processor -- start 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.187+0000 7f3c23fff640 1 -- start start 2024-03-29T11:51:18.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.187+0000 7f3c23fff640 1 librados: setting wanted keys 2024-03-29T11:51:18.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.187+0000 7f3c23fff640 1 librados: calling monclient init 2024-03-29T11:51:18.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.187+0000 7f3c23fff640 10 monclient: init 2024-03-29T11:51:18.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.188+0000 7f3c23fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:18.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.188+0000 7f3c23fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:51:18.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.188+0000 7f3c23fff640 10 monclient(hunting): picked mon.noname-a con 0x7f3c240a9c00 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:18.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.188+0000 7f3c23fff640 10 monclient(hunting): picked mon.noname-c con 0x7f3c240b2800 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c23fff640 10 monclient(hunting): picked mon.noname-b con 0x7f3c240b38c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c23fff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c241da5f0 con 0x7f3c240a9c00 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c237fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3c240b38c0 0x7f3c240b0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:40763/0 (socket says 172.21.15.55:40763) 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c237fe640 1 -- 172.21.15.55:0/2751176740 learned_addr learned my addr 172.21.15.55:0/2751176740 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c0b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c240b38c0 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c240afca0 con 0x7f3c240b38c0 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c241db190 con 0x7f3c240b2800 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c23fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c23fff640 10 monclient(hunting): authenticate will time out at 1491.470451s 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409155370 0 0) 0x7f3c240afca0 con 0x7f3c240b38c0 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:51:18.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.191+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c10004eb0 con 0x7f3c240b38c0 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c240b2800 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616973610 0 0) 0x7f3c241db190 con 0x7f3c240b2800 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c240afca0 con 0x7f3c240b2800 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2400987823 0 0) 0x7f3c10004eb0 con 0x7f3c240b38c0 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c241db190 con 0x7f3c240b38c0 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3c14001750 con 0x7f3c240b38c0 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:18.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient(hunting): dump: 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c240a9c00 2024-03-29T11:51:18.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1830920043 0 0) 0x7f3c241db190 con 0x7f3c240b38c0 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 >> v1:172.21.15.55:6790/0 conn(0x7f3c240b2800 legacy=0x7f3c241daa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 >> v1:172.21.15.55:6789/0 conn(0x7f3c240a9c00 legacy=0x7f3c240ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 monclient: found mon.b 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c241db320 con 0x7f3c240b38c0 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.192+0000 7f3c0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c14001c60 con 0x7f3c240b38c0 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f3c14004860 con 0x7f3c240b38c0 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:18.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c0b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c0b7fe640 10 monclient: dump: 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c23fff640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c23fff640 10 monclient: _renew_subs 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c23fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c240b0410 con 0x7f3c240b38c0 2024-03-29T11:51:18.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c23fff640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c23fff640 10 monclient: _renew_subs 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c23fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c240b0760 con 0x7f3c240b38c0 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.193+0000 7f3c23fff640 1 librados: init done 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.194+0000 7f3c20ff9640 4 set_mon_vals no callback set 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.195+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f3c1401a830 con 0x7f3c240b38c0 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.195+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f3c140847e0 con 0x7f3c240b38c0 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.195+0000 7f3c0b7fe640 10 client.4537.objecter ms_dispatch 0x7f3c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.195+0000 7f3c0b7fe640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.195+0000 7f3c0b7fe640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.195+0000 7f3c0b7fe640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.196+0000 7f3c23fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.196+0000 7f3c23fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.196+0000 7f3c23fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.196+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c04005080 con 0x7f3c240b38c0 2024-03-29T11:51:18.209 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.196+0000 7f3c22ffd640 10 monclient: get_auth_request con 0x7f3c10076570 auth_method 0 2024-03-29T11:51:18.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.196+0000 7f3c0b7fe640 10 client.4537.objecter ms_handle_connect 0x7f3c10076570 2024-03-29T11:51:18.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.205+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c1404fef0 con 0x7f3c240b38c0 2024-03-29T11:51:18.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.205+0000 7f3c0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:18.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.205+0000 7f3c0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:18.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.536+0000 7f3c23fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:18.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.536+0000 7f3c23fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:18.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.536+0000 7f3c23fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:51:18.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.536+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c04005330 con 0x7f3c240b38c0 2024-03-29T11:51:18.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.537+0000 7f3c0b7fe640 1 -- 172.21.15.55:0/2751176740 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f3c14051ff0 con 0x7f3c240b38c0 2024-03-29T11:51:18.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.537+0000 7f3c0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:18.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.537+0000 7f3c0b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:18.543 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:51:18.544 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.540+0000 7f3c23fff640 10 librados: watch_flush enter 2024-03-29T11:51:18.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.540+0000 7f3c23fff640 10 librados: watch_flush exit 2024-03-29T11:51:18.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.540+0000 7f3c23fff640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-29T11:51:18.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.540+0000 7f3c23fff640 10 client.4537.objecter successfully canceled tick 2024-03-29T11:51:18.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.541+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 >> v1:172.21.15.171:6816/1463197163 conn(0x7f3c10076570 legacy=0x7f3c10078a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:18.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.541+0000 7f3c23fff640 10 monclient: shutdown 2024-03-29T11:51:18.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.541+0000 7f3c23fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:18.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.541+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 >> v1:172.21.15.171:6789/0 conn(0x7f3c240b38c0 legacy=0x7f3c240b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:18.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.542+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 shutdown_connections 2024-03-29T11:51:18.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.542+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 >> 172.21.15.55:0/2751176740 conn(0x7f3c24066a00 msgr2=0x7f3c2413bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:18.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.542+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 shutdown_connections 2024-03-29T11:51:18.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.542+0000 7f3c23fff640 1 -- 172.21.15.55:0/2751176740 wait complete. 2024-03-29T11:51:18.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:18.542+0000 7f3c23fff640 1 librados: shutdown 2024-03-29T11:51:18.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:25.374 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.519+0000 7fe9ef393640 10 monclient: get_monmap_and_config 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.519+0000 7fe9ef393640 10 monclient: build_initial_monmap 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.519+0000 7fe9ef393640 10 monclient: monmap: 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:51:25.520680+0000 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:51:25.520680+0000 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:51:25.528 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:25.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.522+0000 7fe9ef393640 1 Processor -- start 2024-03-29T11:51:25.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.523+0000 7fe9ef393640 1 -- start start 2024-03-29T11:51:25.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.523+0000 7fe9ef393640 10 monclient: init 2024-03-29T11:51:25.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.524+0000 7fe9ef393640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:25.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.524+0000 7fe9ef393640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:51:25.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.524+0000 7fe9ef393640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e80ae250 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:25.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.528+0000 7fe9ef393640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e813f970 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:25.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.528+0000 7fe9ef393640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e8142800 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:25.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.528+0000 7fe9ef393640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80afe00 con 0x7fe9e80ae250 2024-03-29T11:51:25.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.528+0000 7fe9ef393640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e8142c10 con 0x7fe9e8142800 2024-03-29T11:51:25.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.528+0000 7fe9ef393640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80afb20 con 0x7fe9e813f970 2024-03-29T11:51:25.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.528+0000 7fe9ef393640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:25.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.528+0000 7fe9ef393640 10 monclient(hunting): authenticate will time out at 1498.807826s 2024-03-29T11:51:25.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.528+0000 7fe9ec92e640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fe9e813f970 0x7fe9e81420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:60450/0 (socket says 172.21.15.55:60450) 2024-03-29T11:51:25.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.528+0000 7fe9ec92e640 1 -- 172.21.15.55:0/4063545134 learned_addr learned my addr 172.21.15.55:0/4063545134 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:25.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.529+0000 7fe9dffff640 1 -- 172.21.15.55:0/4063545134 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319145033 0 0) 0x7fe9e80afb20 con 0x7fe9e813f970 2024-03-29T11:51:25.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.529+0000 7fe9dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:51:25.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.529+0000 7fe9dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:25.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.529+0000 7fe9dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:25.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.529+0000 7fe9dffff640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:51:25.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.529+0000 7fe9dffff640 1 -- 172.21.15.55:0/4063545134 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d4003520 con 0x7fe9e813f970 2024-03-29T11:51:25.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.529+0000 7fe9dffff640 1 -- 172.21.15.55:0/4063545134 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 200059956 0 0) 0x7fe9d4003520 con 0x7fe9e813f970 2024-03-29T11:51:25.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.529+0000 7fe9dffff640 1 -- 172.21.15.55:0/4063545134 >> v1:172.21.15.171:6789/0 conn(0x7fe9e8142800 legacy=0x7fe9e814da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:51:25.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.532+0000 7fe9dffff640 1 -- 172.21.15.55:0/4063545134 >> v1:172.21.15.55:6789/0 conn(0x7fe9e80ae250 legacy=0x7fe9e813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.532+0000 7fe9dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.532+0000 7fe9dffff640 1 monclient: found mon.noname-b 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.532+0000 7fe9dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 1 -- 172.21.15.55:0/4063545134 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80aab80 con 0x7fe9e813f970 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: _finish_auth 0 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 1 -- 172.21.15.55:0/4063545134 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe9d8001670 con 0x7fe9e813f970 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: dump: 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.540 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.541 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:25.541 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:25.541 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:25.541 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:25.541 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:25.541 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:25.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9ef393640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:51:25.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9ef393640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 1 -- 172.21.15.55:0/4063545134 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9d80018d0 con 0x7fe9e813f970 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 1 -- 172.21.15.55:0/4063545134 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe9d8001d20 con 0x7fe9e813f970 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9dffff640 10 monclient: dump: 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:25.542 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:25.543 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:25.543 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9ef393640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.533+0000 7fe9ef393640 4 set_mon_vals no callback set 2024-03-29T11:51:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.534+0000 7fe9ef393640 10 monclient: shutdown 2024-03-29T11:51:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.534+0000 7fe9ef393640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.534+0000 7fe9ef393640 1 -- 172.21.15.55:0/4063545134 >> v1:172.21.15.55:6790/0 conn(0x7fe9e813f970 legacy=0x7fe9e81420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.535+0000 7fe9ef393640 1 -- 172.21.15.55:0/4063545134 shutdown_connections 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.535+0000 7fe9ef393640 1 -- 172.21.15.55:0/4063545134 >> 172.21.15.55:0/4063545134 conn(0x7fe9e80b35e0 msgr2=0x7fe9e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.536+0000 7fe9ef393640 1 -- 172.21.15.55:0/4063545134 shutdown_connections 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.536+0000 7fe9ef393640 1 -- 172.21.15.55:0/4063545134 wait complete. 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.537+0000 7fe9ef393640 10 monclient: build_initial_monmap 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.537+0000 7fe9ef393640 10 monclient: monmap: 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:51:25.544 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:51:25.545 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:25.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.537+0000 7fe9ef393640 1 librados: starting msgr at 2024-03-29T11:51:25.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.538+0000 7fe9ef393640 1 librados: starting objecter 2024-03-29T11:51:25.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.538+0000 7fe9ef393640 1 Processor -- start 2024-03-29T11:51:25.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.538+0000 7fe9ef393640 1 -- start start 2024-03-29T11:51:25.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.538+0000 7fe9ef393640 1 librados: setting wanted keys 2024-03-29T11:51:25.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.538+0000 7fe9ef393640 1 librados: calling monclient init 2024-03-29T11:51:25.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.538+0000 7fe9ef393640 10 monclient: init 2024-03-29T11:51:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.539+0000 7fe9ef393640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.539+0000 7fe9ef393640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:51:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.539+0000 7fe9ef393640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e80ae250 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.540+0000 7fe9ef393640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e813f970 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.540+0000 7fe9ef393640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e8142800 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:25.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.540+0000 7fe9ef393640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e813ba00 con 0x7fe9e80ae250 2024-03-29T11:51:25.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.540+0000 7fe9ef393640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e8144c40 con 0x7fe9e8142800 2024-03-29T11:51:25.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.540+0000 7fe9ef393640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e81d7590 con 0x7fe9e813f970 2024-03-29T11:51:25.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.540+0000 7fe9ef393640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:25.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.540+0000 7fe9ef393640 10 monclient(hunting): authenticate will time out at 1498.819973s 2024-03-29T11:51:25.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.542+0000 7fe9ec92e640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fe9e813f970 0x7fe9e81d6e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:34312/0 (socket says 172.21.15.55:34312) 2024-03-29T11:51:25.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.542+0000 7fe9ec92e640 1 -- 172.21.15.55:0/2497234876 learned_addr learned my addr 172.21.15.55:0/2497234876 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:25.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.542+0000 7fe9ddffb640 10 client.?.objecter ms_handle_connect 0x7fe9e813f970 2024-03-29T11:51:25.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.542+0000 7fe9ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:25.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.542+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047783838 0 0) 0x7fe9e81d7590 con 0x7fe9e813f970 2024-03-29T11:51:25.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:51:25.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:25.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:25.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:51:25.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9cc004eb0 con 0x7fe9e813f970 2024-03-29T11:51:25.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3365111207 0 0) 0x7fe9cc004eb0 con 0x7fe9e813f970 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9e81d7590 con 0x7fe9e813f970 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe9d8001d20 con 0x7fe9e813f970 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 monclient(hunting): dump: 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:25.555 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 client.?.objecter ms_handle_connect 0x7fe9e80ae250 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.543+0000 7fe9ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784656316 0 0) 0x7fe9e813ba00 con 0x7fe9e80ae250 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9cc004eb0 con 0x7fe9e80ae250 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 36393955 0 0) 0x7fe9cc004eb0 con 0x7fe9e80ae250 2024-03-29T11:51:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9cc0059c0 con 0x7fe9e80ae250 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe9e40018b0 con 0x7fe9e80ae250 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient(hunting): dump: 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:25.557 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 931869289 0 0) 0x7fe9cc0059c0 con 0x7fe9e80ae250 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 >> v1:172.21.15.55:6790/0 conn(0x7fe9e813f970 legacy=0x7fe9e81d6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 >> v1:172.21.15.171:6789/0 conn(0x7fe9e8142800 legacy=0x7fe9e81e1790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 monclient: found mon.a 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e8143470 con 0x7fe9e80ae250 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9e4001d40 con 0x7fe9e80ae250 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fe9e40018b0 con 0x7fe9e80ae250 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ddffb640 10 monclient: dump: 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ef393640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.544+0000 7fe9ef393640 10 monclient: _renew_subs 2024-03-29T11:51:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.545+0000 7fe9ef393640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.545+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e8146210 con 0x7fe9e80ae250 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.545+0000 7fe9ef393640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.545+0000 7fe9ef393640 10 monclient: _renew_subs 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.545+0000 7fe9ef393640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.545+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e8143a00 con 0x7fe9e80ae250 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.545+0000 7fe9ef393640 1 librados: init done 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.545+0000 7fe9df7fe640 4 set_mon_vals no callback set 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.547+0000 7fe9ef393640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.547+0000 7fe9ef393640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.547+0000 7fe9ef393640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.547+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e8145ed0 con 0x7fe9e80ae250 2024-03-29T11:51:25.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.549+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fe9e40194e0 con 0x7fe9e80ae250 2024-03-29T11:51:25.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.550+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fe9e40831a0 con 0x7fe9e80ae250 2024-03-29T11:51:25.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.550+0000 7fe9ddffb640 10 client.4569.objecter ms_dispatch 0x7fe9e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:25.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.550+0000 7fe9ddffb640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:25.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.550+0000 7fe9ddffb640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:25.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.550+0000 7fe9ddffb640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-29T11:51:25.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.551+0000 7fe9ec92e640 10 monclient: get_auth_request con 0x7fe9cc06e330 auth_method 0 2024-03-29T11:51:25.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.554+0000 7fe9ddffb640 10 client.4569.objecter ms_handle_connect 0x7fe9cc06e330 2024-03-29T11:51:25.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.555+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9e404e8b0 con 0x7fe9e80ae250 2024-03-29T11:51:25.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.555+0000 7fe9ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:25.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.555+0000 7fe9ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.890+0000 7fe9ef393640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.890+0000 7fe9ef393640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.890+0000 7fe9ef393640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.890+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9e8075f10 con 0x7fe9e80ae250 2024-03-29T11:51:25.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.894+0000 7fe9ddffb640 1 -- 172.21.15.55:0/2497234876 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fe9e404e8b0 con 0x7fe9e80ae250 2024-03-29T11:51:25.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.894+0000 7fe9ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:25.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.894+0000 7fe9ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:25.904 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:51:25.904 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 10 librados: watch_flush enter 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 10 librados: watch_flush exit 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 10 client.4569.objecter successfully canceled tick 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 >> v1:172.21.15.171:6816/1463197163 conn(0x7fe9cc06e330 legacy=0x7fe9cc0707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 10 monclient: shutdown 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 >> v1:172.21.15.55:6789/0 conn(0x7fe9e80ae250 legacy=0x7fe9e8144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 shutdown_connections 2024-03-29T11:51:25.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.900+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 >> 172.21.15.55:0/2497234876 conn(0x7fe9e80b35e0 msgr2=0x7fe9e80b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:25.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.904+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 shutdown_connections 2024-03-29T11:51:25.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.905+0000 7fe9ef393640 1 -- 172.21.15.55:0/2497234876 wait complete. 2024-03-29T11:51:25.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:25.905+0000 7fe9ef393640 1 librados: shutdown 2024-03-29T11:51:25.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:32.741 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:32.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.883+0000 7f20d0fb5640 10 monclient: get_monmap_and_config 2024-03-29T11:51:32.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.883+0000 7f20d0fb5640 10 monclient: build_initial_monmap 2024-03-29T11:51:32.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.883+0000 7f20d0fb5640 10 monclient: monmap: 2024-03-29T11:51:32.890 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:32.890 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:32.890 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:51:32.884388+0000 2024-03-29T11:51:32.890 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:51:32.884388+0000 2024-03-29T11:51:32.891 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:32.891 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:32.891 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:32.891 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:51:32.891 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:51:32.891 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:32.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.884+0000 7f20d0fb5640 1 Processor -- start 2024-03-29T11:51:32.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.884+0000 7f20d0fb5640 1 -- start start 2024-03-29T11:51:32.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.884+0000 7f20d0fb5640 10 monclient: init 2024-03-29T11:51:32.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:32.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:51:32.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f20cc1422b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:32.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f20cc142690 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:32.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f20cc0b17d0 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:32.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc142c80 con 0x7f20cc142690 2024-03-29T11:51:32.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc0aae40 con 0x7f20cc0b17d0 2024-03-29T11:51:32.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc13fbd0 con 0x7f20cc1422b0 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.885+0000 7f20d0fb5640 10 monclient(hunting): authenticate will time out at 1506.165274s 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c9d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f20cc142690 0x7f20cc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46992/0 (socket says 172.21.15.55:46992) 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c9d9b640 1 -- 172.21.15.55:0/2746797678 learned_addr learned my addr 172.21.15.55:0/2746797678 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302865181 0 0) 0x7f20cc142c80 con 0x7f20cc142690 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b4003520 con 0x7f20cc142690 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718030965 0 0) 0x7f20cc0aae40 con 0x7f20cc0b17d0 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:32.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b4003840 con 0x7f20cc0b17d0 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581180671 0 0) 0x7f20cc13fbd0 con 0x7f20cc1422b0 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.886+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b4003b80 con 0x7f20cc1422b0 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2769077738 0 0) 0x7f20b4003b80 con 0x7f20cc1422b0 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 >> v1:172.21.15.171:6789/0 conn(0x7f20cc0b17d0 legacy=0x7f20cc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 >> v1:172.21.15.55:6789/0 conn(0x7f20cc142690 legacy=0x7f20cc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 10 monclient: _finish_hunting 0 2024-03-29T11:51:32.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 1 monclient: found mon.noname-b 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc151940 con 0x7f20cc1422b0 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 10 monclient: _finish_auth 0 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 10 monclient: _check_auth_tickets 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20d0fb5640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20d0fb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f20bc001630 con 0x7f20cc1422b0 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 10 monclient: dump: 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:32.899 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20d0fb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20bc001810 con 0x7f20cc1422b0 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20c959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20d0fb5640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.887+0000 7f20d0fb5640 4 set_mon_vals no callback set 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20c959a640 1 -- 172.21.15.55:0/2746797678 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f20bc001630 con 0x7f20cc1422b0 2024-03-29T11:51:32.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20c959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20c959a640 10 monclient: dump: 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20d0fb5640 10 monclient: shutdown 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20d0fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:32.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20d0fb5640 1 -- 172.21.15.55:0/2746797678 >> v1:172.21.15.55:6790/0 conn(0x7f20cc1422b0 legacy=0x7f20cc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:32.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20d0fb5640 1 -- 172.21.15.55:0/2746797678 shutdown_connections 2024-03-29T11:51:32.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20d0fb5640 1 -- 172.21.15.55:0/2746797678 >> 172.21.15.55:0/2746797678 conn(0x7f20cc066a00 msgr2=0x7f20cc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:32.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20d0fb5640 1 -- 172.21.15.55:0/2746797678 shutdown_connections 2024-03-29T11:51:32.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.888+0000 7f20d0fb5640 1 -- 172.21.15.55:0/2746797678 wait complete. 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.890+0000 7f20d0fb5640 10 monclient: build_initial_monmap 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.890+0000 7f20d0fb5640 10 monclient: monmap: 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.891+0000 7f20d0fb5640 1 librados: starting msgr at 2024-03-29T11:51:32.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.891+0000 7f20d0fb5640 1 librados: starting objecter 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.891+0000 7f20d0fb5640 1 Processor -- start 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.891+0000 7f20d0fb5640 1 -- start start 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.891+0000 7f20d0fb5640 1 librados: setting wanted keys 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.891+0000 7f20d0fb5640 1 librados: calling monclient init 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.891+0000 7f20d0fb5640 10 monclient: init 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f20cc0b17d0 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f20cc1422b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f20cc142690 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc06eef0 con 0x7f20cc0b17d0 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc131920 con 0x7f20cc142690 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc131540 con 0x7f20cc1422b0 2024-03-29T11:51:32.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20d0fb5640 10 monclient(hunting): authenticate will time out at 1506.172147s 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20ca59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f20cc0b17d0 0x7f20cc1dee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47526/0 (socket says 172.21.15.55:47526) 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.892+0000 7f20ca59c640 1 -- 172.21.15.55:0/430617464 learned_addr learned my addr 172.21.15.55:0/430617464 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 10 client.?.objecter ms_handle_connect 0x7f20cc0b17d0 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327067404 0 0) 0x7f20cc06eef0 con 0x7f20cc0b17d0 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ac004eb0 con 0x7f20cc0b17d0 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 10 client.?.objecter ms_handle_connect 0x7f20cc1422b0 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.893+0000 7f20baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418053946 0 0) 0x7f20cc131540 con 0x7f20cc1422b0 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:32.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20cc06eef0 con 0x7f20cc1422b0 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1274783967 0 0) 0x7f20ac004eb0 con 0x7f20cc0b17d0 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20cc131540 con 0x7f20cc0b17d0 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f20bc001d40 con 0x7f20cc0b17d0 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 10 monclient(hunting): dump: 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:32.912 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 852902065 0 0) 0x7f20cc131540 con 0x7f20cc0b17d0 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.894+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 >> v1:172.21.15.55:6790/0 conn(0x7f20cc1422b0 legacy=0x7f20cc12ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 >> v1:172.21.15.171:6789/0 conn(0x7f20cc142690 legacy=0x7f20cc1e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 1 monclient: found mon.a 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc1316d0 con 0x7f20cc0b17d0 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 10 monclient: _finish_auth 0 2024-03-29T11:51:32.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20d0fb5640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20d0fb5640 10 monclient: _renew_subs 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20d0fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc06de90 con 0x7f20cc0b17d0 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20d0fb5640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20d0fb5640 10 monclient: _renew_subs 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20d0fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc132680 con 0x7f20cc0b17d0 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20d0fb5640 1 librados: init done 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20bc0038a0 con 0x7f20cc0b17d0 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.895+0000 7f20c8d99640 4 set_mon_vals no callback set 2024-03-29T11:51:32.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.896+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f20bc003c30 con 0x7f20cc0b17d0 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.896+0000 7f20baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.896+0000 7f20baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.896+0000 7f20baffd640 10 monclient: dump: 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.897+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f20bc003d90 con 0x7f20cc0b17d0 2024-03-29T11:51:32.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20d0fb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20d0fb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20d0fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b0005080 con 0x7f20cc0b17d0 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20c9d9b640 10 monclient: get_auth_request con 0x7f20ac06dd20 auth_method 0 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f20bc084bc0 con 0x7f20cc0b17d0 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20baffd640 10 client.4575.objecter ms_dispatch 0x7f20cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20baffd640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20baffd640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20baffd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-29T11:51:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.898+0000 7f20baffd640 10 client.4575.objecter ms_handle_connect 0x7f20ac06dd20 2024-03-29T11:51:32.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.906+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20bc050500 con 0x7f20cc0b17d0 2024-03-29T11:51:32.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.906+0000 7f20baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:32.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:32.906+0000 7f20baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:33.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.277+0000 7f20d0fb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:33.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.277+0000 7f20d0fb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:33.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.277+0000 7f20d0fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:33.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.277+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20b0005330 con 0x7f20cc0b17d0 2024-03-29T11:51:33.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.279+0000 7f20baffd640 1 -- 172.21.15.55:0/430617464 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f20bc084270 con 0x7f20cc0b17d0 2024-03-29T11:51:33.286 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:51:33.286 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:33.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.279+0000 7f20baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:33.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.279+0000 7f20baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:33.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.283+0000 7f20d0fb5640 10 librados: watch_flush enter 2024-03-29T11:51:33.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.283+0000 7f20d0fb5640 10 librados: watch_flush exit 2024-03-29T11:51:33.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.283+0000 7f20d0fb5640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-29T11:51:33.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.283+0000 7f20d0fb5640 10 client.4575.objecter successfully canceled tick 2024-03-29T11:51:33.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.283+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 >> v1:172.21.15.171:6816/1463197163 conn(0x7f20ac06dd20 legacy=0x7f20ac0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:33.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.283+0000 7f20d0fb5640 10 monclient: shutdown 2024-03-29T11:51:33.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.283+0000 7f20d0fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:33.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.283+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 >> v1:172.21.15.55:6789/0 conn(0x7f20cc0b17d0 legacy=0x7f20cc1dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:33.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.284+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 shutdown_connections 2024-03-29T11:51:33.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.284+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 >> 172.21.15.55:0/430617464 conn(0x7f20cc066a00 msgr2=0x7f20cc13dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:33.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.284+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 shutdown_connections 2024-03-29T11:51:33.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.284+0000 7f20d0fb5640 1 -- 172.21.15.55:0/430617464 wait complete. 2024-03-29T11:51:33.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:33.284+0000 7f20d0fb5640 1 librados: shutdown 2024-03-29T11:51:33.310 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:40.116 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:40.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.270+0000 7fdf58f96640 10 monclient: get_monmap_and_config 2024-03-29T11:51:40.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.270+0000 7fdf58f96640 10 monclient: build_initial_monmap 2024-03-29T11:51:40.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.270+0000 7fdf58f96640 10 monclient: monmap: 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:51:40.271694+0000 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:51:40.271694+0000 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:51:40.278 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:40.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.271+0000 7fdf58f96640 1 Processor -- start 2024-03-29T11:51:40.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.271+0000 7fdf58f96640 1 -- start start 2024-03-29T11:51:40.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.271+0000 7fdf58f96640 10 monclient: init 2024-03-29T11:51:40.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:40.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:51:40.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 10 monclient(hunting): picked mon.noname-b con 0x7fdf541422b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:40.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 10 monclient(hunting): picked mon.noname-a con 0x7fdf54142690 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:40.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 10 monclient(hunting): picked mon.noname-c con 0x7fdf540b17d0 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:40.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf54142c80 con 0x7fdf54142690 2024-03-29T11:51:40.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540aae40 con 0x7fdf540b17d0 2024-03-29T11:51:40.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5413fbd0 con 0x7fdf541422b0 2024-03-29T11:51:40.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:40.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf58f96640 10 monclient(hunting): authenticate will time out at 1513.552126s 2024-03-29T11:51:40.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf5259c640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fdf541422b0 0x7fdf5413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:35862/0 (socket says 172.21.15.55:35862) 2024-03-29T11:51:40.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.272+0000 7fdf5259c640 1 -- 172.21.15.55:0/871901842 learned_addr learned my addr 172.21.15.55:0/871901842 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916446200 0 0) 0x7fdf54142c80 con 0x7fdf54142690 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf3c003520 con 0x7fdf54142690 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898331644 0 0) 0x7fdf5413fbd0 con 0x7fdf541422b0 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:40.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf54142c80 con 0x7fdf541422b0 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699440584 0 0) 0x7fdf540aae40 con 0x7fdf540b17d0 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5413fbd0 con 0x7fdf540b17d0 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 314030948 0 0) 0x7fdf3c003520 con 0x7fdf54142690 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 >> v1:172.21.15.55:6790/0 conn(0x7fdf541422b0 legacy=0x7fdf5413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 >> v1:172.21.15.171:6789/0 conn(0x7fdf540b17d0 legacy=0x7fdf540b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient: _finish_hunting 0 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 monclient: found mon.noname-a 2024-03-29T11:51:40.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf54151940 con 0x7fdf54142690 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient: _finish_auth 0 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 10 monclient: _check_auth_tickets 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.273+0000 7fdf5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdf48001690 con 0x7fdf54142690 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf5159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf5159a640 10 monclient: dump: 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:40.285 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf58f96640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf58f96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf48001870 con 0x7fdf54142690 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf5159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf58f96640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.274+0000 7fdf58f96640 4 set_mon_vals no callback set 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf5159a640 1 -- 172.21.15.55:0/871901842 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdf48001c00 con 0x7fdf54142690 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf5159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:40.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf5159a640 10 monclient: dump: 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf58f96640 10 monclient: shutdown 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf58f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf58f96640 1 -- 172.21.15.55:0/871901842 >> v1:172.21.15.55:6789/0 conn(0x7fdf54142690 legacy=0x7fdf540b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf58f96640 1 -- 172.21.15.55:0/871901842 shutdown_connections 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf58f96640 1 -- 172.21.15.55:0/871901842 >> 172.21.15.55:0/871901842 conn(0x7fdf54066a00 msgr2=0x7fdf54066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:40.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.275+0000 7fdf58f96640 1 -- 172.21.15.55:0/871901842 shutdown_connections 2024-03-29T11:51:40.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.276+0000 7fdf58f96640 1 -- 172.21.15.55:0/871901842 wait complete. 2024-03-29T11:51:40.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.277+0000 7fdf58f96640 10 monclient: build_initial_monmap 2024-03-29T11:51:40.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.277+0000 7fdf58f96640 10 monclient: monmap: 2024-03-29T11:51:40.294 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:40.294 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:40.294 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:51:40.294 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:51:40.294 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:40.294 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:40.294 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.277+0000 7fdf58f96640 1 librados: starting msgr at 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.277+0000 7fdf58f96640 1 librados: starting objecter 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.278+0000 7fdf58f96640 1 Processor -- start 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.278+0000 7fdf58f96640 1 -- start start 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.278+0000 7fdf58f96640 1 librados: setting wanted keys 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.278+0000 7fdf58f96640 1 librados: calling monclient init 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.278+0000 7fdf58f96640 10 monclient: init 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 10 monclient(hunting): picked mon.noname-b con 0x7fdf540b17d0 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:40.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 10 monclient(hunting): picked mon.noname-c con 0x7fdf541422b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 10 monclient(hunting): picked mon.noname-a con 0x7fdf54142690 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf541db8a0 con 0x7fdf54142690 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf541dbb80 con 0x7fdf540b17d0 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf541dc720 con 0x7fdf541422b0 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf58f96640 10 monclient(hunting): authenticate will time out at 1513.558799s 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf52d9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fdf54142690 0x7fdf5413bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:34498/0 (socket says 172.21.15.55:34498) 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf52d9d640 1 -- 172.21.15.55:0/3727263489 learned_addr learned my addr 172.21.15.55:0/3727263489 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 client.?.objecter ms_handle_connect 0x7fdf54142690 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 client.?.objecter ms_handle_connect 0x7fdf541422b0 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 client.?.objecter ms_handle_connect 0x7fdf540b17d0 2024-03-29T11:51:40.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773281478 0 0) 0x7fdf541db8a0 con 0x7fdf54142690 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.279+0000 7fdf42ffd640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf34004eb0 con 0x7fdf54142690 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102726109 0 0) 0x7fdf541dc720 con 0x7fdf541422b0 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf541db8a0 con 0x7fdf541422b0 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2550567410 0 0) 0x7fdf34004eb0 con 0x7fdf54142690 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf541dc720 con 0x7fdf54142690 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdf4c001c80 con 0x7fdf54142690 2024-03-29T11:51:40.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): dump: 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539990058 0 0) 0x7fdf541dbb80 con 0x7fdf540b17d0 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:40.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.280+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf34004eb0 con 0x7fdf540b17d0 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.281+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3675961659 0 0) 0x7fdf541db8a0 con 0x7fdf541422b0 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.281+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf541dbb80 con 0x7fdf541422b0 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.281+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdf48001b80 con 0x7fdf541422b0 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.281+0000 7fdf42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.281+0000 7fdf42ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.281+0000 7fdf42ffd640 10 monclient(hunting): dump: 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:40.299 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.281+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 127553759 0 0) 0x7fdf541dbb80 con 0x7fdf541422b0 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.281+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 >> v1:172.21.15.171:6789/0 conn(0x7fdf540b17d0 legacy=0x7fdf54071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.281+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 >> v1:172.21.15.55:6789/0 conn(0x7fdf54142690 legacy=0x7fdf5413bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 1 monclient: found mon.c 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf54070fc0 con 0x7fdf541422b0 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:51:40.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf58f96640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf58f96640 10 monclient: _renew_subs 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf58f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf540712a0 con 0x7fdf541422b0 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf58f96640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf58f96640 10 monclient: _renew_subs 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf58f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5406ec40 con 0x7fdf541422b0 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf58f96640 1 librados: init done 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf48001d40 con 0x7fdf541422b0 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf43fff640 4 set_mon_vals no callback set 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fdf48003a40 con 0x7fdf541422b0 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:40.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.282+0000 7fdf42ffd640 10 monclient: dump: 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.283+0000 7fdf58f96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.283+0000 7fdf58f96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.283+0000 7fdf58f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.283+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf38005080 con 0x7fdf541422b0 2024-03-29T11:51:40.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.284+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fdf48019a10 con 0x7fdf541422b0 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.285+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fdf48083890 con 0x7fdf541422b0 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.285+0000 7fdf5259c640 10 monclient: get_auth_request con 0x7fdf3406e050 auth_method 0 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.285+0000 7fdf42ffd640 10 client.4595.objecter ms_dispatch 0x7fdf54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.285+0000 7fdf42ffd640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.285+0000 7fdf42ffd640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.285+0000 7fdf42ffd640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.285+0000 7fdf42ffd640 10 client.4595.objecter ms_handle_connect 0x7fdf3406e050 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.294+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf4804f170 con 0x7fdf541422b0 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.294+0000 7fdf42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:40.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.294+0000 7fdf42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:40.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.684+0000 7fdf58f96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:40.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.684+0000 7fdf58f96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:40.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.684+0000 7fdf58f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:40.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.684+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf38005330 con 0x7fdf541422b0 2024-03-29T11:51:40.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.686+0000 7fdf42ffd640 1 -- 172.21.15.55:0/3727263489 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fdf480510a0 con 0x7fdf541422b0 2024-03-29T11:51:40.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.686+0000 7fdf42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:40.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.686+0000 7fdf42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:40.693 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:51:40.693 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:40.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 10 librados: watch_flush enter 2024-03-29T11:51:40.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 10 librados: watch_flush exit 2024-03-29T11:51:40.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T11:51:40.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 10 client.4595.objecter successfully canceled tick 2024-03-29T11:51:40.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 >> v1:172.21.15.171:6816/1463197163 conn(0x7fdf3406e050 legacy=0x7fdf34070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:40.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 10 monclient: shutdown 2024-03-29T11:51:40.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:40.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 >> v1:172.21.15.55:6790/0 conn(0x7fdf541422b0 legacy=0x7fdf541dc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:40.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 shutdown_connections 2024-03-29T11:51:40.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 >> 172.21.15.55:0/3727263489 conn(0x7fdf54066a00 msgr2=0x7fdf5413e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:40.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 shutdown_connections 2024-03-29T11:51:40.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 1 -- 172.21.15.55:0/3727263489 wait complete. 2024-03-29T11:51:40.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:40.690+0000 7fdf58f96640 1 librados: shutdown 2024-03-29T11:51:40.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:47.518 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:47.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.662+0000 7faa0759e640 10 monclient: get_monmap_and_config 2024-03-29T11:51:47.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.662+0000 7faa0759e640 10 monclient: build_initial_monmap 2024-03-29T11:51:47.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.662+0000 7faa0759e640 10 monclient: monmap: 2024-03-29T11:51:47.670 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:47.670 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:47.670 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:51:47.663773+0000 2024-03-29T11:51:47.670 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:51:47.663773+0000 2024-03-29T11:51:47.670 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:47.671 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:47.671 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:47.671 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:51:47.671 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:51:47.671 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:47.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.664+0000 7faa0759e640 1 Processor -- start 2024-03-29T11:51:47.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.664+0000 7faa0759e640 1 -- start start 2024-03-29T11:51:47.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.664+0000 7faa0759e640 10 monclient: init 2024-03-29T11:51:47.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.665+0000 7faa0759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:47.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.665+0000 7faa0759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:51:47.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.665+0000 7faa0759e640 10 monclient(hunting): picked mon.noname-c con 0x7faa00006a70 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.668+0000 7faa0659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7faa00006a70 0x7faa000c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:35543/0 (socket says 172.21.15.55:35543) 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.668+0000 7faa0659c640 1 -- 172.21.15.55:0/1866543526 learned_addr learned my addr 172.21.15.55:0/1866543526 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.668+0000 7faa0759e640 10 monclient(hunting): picked mon.noname-b con 0x7faa0002b100 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.669+0000 7faa0759e640 10 monclient(hunting): picked mon.noname-a con 0x7faa00029c30 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.669+0000 7faa0759e640 1 -- 172.21.15.55:0/1866543526 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0002a270 con 0x7faa00029c30 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.669+0000 7faa0759e640 1 -- 172.21.15.55:0/1866543526 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0002c440 con 0x7faa00006a70 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.669+0000 7faa0759e640 1 -- 172.21.15.55:0/1866543526 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa000c9440 con 0x7faa0002b100 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.669+0000 7faa0759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.669+0000 7faa0759e640 10 monclient(hunting): authenticate will time out at 1520.949558s 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.670+0000 7faa0559a640 1 -- 172.21.15.55:0/1866543526 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940070789 0 0) 0x7faa000c9440 con 0x7faa0002b100 2024-03-29T11:51:47.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.670+0000 7faa0559a640 10 monclient(hunting): _negotiate 2024-03-29T11:51:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.670+0000 7faa0559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.670+0000 7faa0559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.670+0000 7faa0559a640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:51:47.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.670+0000 7faa0559a640 1 -- 172.21.15.55:0/1866543526 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f0003520 con 0x7faa0002b100 2024-03-29T11:51:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.672+0000 7faa0559a640 1 -- 172.21.15.55:0/1866543526 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3721034520 0 0) 0x7fa9f0003520 con 0x7faa0002b100 2024-03-29T11:51:47.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.672+0000 7faa0559a640 1 -- 172.21.15.55:0/1866543526 >> v1:172.21.15.171:6789/0 conn(0x7faa00006a70 legacy=0x7faa000c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.672+0000 7faa0559a640 1 -- 172.21.15.55:0/1866543526 >> v1:172.21.15.55:6789/0 conn(0x7faa00029c30 legacy=0x7faa000d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.672+0000 7faa0559a640 10 monclient: _finish_hunting 0 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.672+0000 7faa0559a640 1 monclient: found mon.noname-b 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.672+0000 7faa0559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.672+0000 7faa0559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.672+0000 7faa0559a640 1 -- 172.21.15.55:0/1866543526 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa000c95d0 con 0x7faa0002b100 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 10 monclient: _finish_auth 0 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 10 monclient: _check_auth_tickets 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 1 -- 172.21.15.55:0/1866543526 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa9fc001690 con 0x7faa0002b100 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0759e640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:47.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 10 monclient: dump: 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 1 -- 172.21.15.55:0/1866543526 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9fc001920 con 0x7faa0002b100 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 1 -- 172.21.15.55:0/1866543526 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa9fc001cb0 con 0x7faa0002b100 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:47.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0759e640 4 set_mon_vals no callback set 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.673+0000 7faa0559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.674+0000 7faa0559a640 10 monclient: dump: 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.674+0000 7faa0759e640 10 monclient: shutdown 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.674+0000 7faa0759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:47.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.674+0000 7faa0759e640 1 -- 172.21.15.55:0/1866543526 >> v1:172.21.15.55:6790/0 conn(0x7faa0002b100 legacy=0x7faa000cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.676+0000 7faa0759e640 1 -- 172.21.15.55:0/1866543526 shutdown_connections 2024-03-29T11:51:47.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.676+0000 7faa0759e640 1 -- 172.21.15.55:0/1866543526 >> 172.21.15.55:0/1866543526 conn(0x7faa000058f0 msgr2=0x7faa00005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:47.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.680+0000 7faa0759e640 1 -- 172.21.15.55:0/1866543526 shutdown_connections 2024-03-29T11:51:47.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.682+0000 7faa0759e640 1 -- 172.21.15.55:0/1866543526 wait complete. 2024-03-29T11:51:47.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.683+0000 7faa0759e640 10 monclient: build_initial_monmap 2024-03-29T11:51:47.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.683+0000 7faa0759e640 10 monclient: monmap: 2024-03-29T11:51:47.690 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:47.690 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:47.690 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:51:47.690 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:51:47.690 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:47.690 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:47.691 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:47.691 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:51:47.691 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:51:47.691 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:47.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.683+0000 7faa0759e640 1 librados: starting msgr at 2024-03-29T11:51:47.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.686+0000 7faa0759e640 1 librados: starting objecter 2024-03-29T11:51:47.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.686+0000 7faa0759e640 1 Processor -- start 2024-03-29T11:51:47.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.688+0000 7faa0759e640 1 -- start start 2024-03-29T11:51:47.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.688+0000 7faa0759e640 1 librados: setting wanted keys 2024-03-29T11:51:47.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.688+0000 7faa0759e640 1 librados: calling monclient init 2024-03-29T11:51:47.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.688+0000 7faa0759e640 10 monclient: init 2024-03-29T11:51:47.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:47.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:51:47.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 10 monclient(hunting): picked mon.noname-a con 0x7faa00006a70 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:47.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 10 monclient(hunting): picked mon.noname-c con 0x7faa00029c30 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:47.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 10 monclient(hunting): picked mon.noname-b con 0x7faa000cd9b0 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:47.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa000ca290 con 0x7faa00006a70 2024-03-29T11:51:47.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa000cca50 con 0x7faa000cd9b0 2024-03-29T11:51:47.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa000cde00 con 0x7faa00029c30 2024-03-29T11:51:47.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:47.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.689+0000 7faa0759e640 10 monclient(hunting): authenticate will time out at 1520.969407s 2024-03-29T11:51:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.692+0000 7faa0659c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7faa00006a70 0x7faa000bf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:39709/0 (socket says 172.21.15.55:39709) 2024-03-29T11:51:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.692+0000 7faa0659c640 1 -- 172.21.15.55:0/43386796 learned_addr learned my addr 172.21.15.55:0/43386796 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.693+0000 7fa9f6ffd640 10 client.?.objecter ms_handle_connect 0x7faa00006a70 2024-03-29T11:51:47.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.693+0000 7fa9f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.693+0000 7fa9f6ffd640 10 client.?.objecter ms_handle_connect 0x7faa000cd9b0 2024-03-29T11:51:47.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.693+0000 7fa9f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.695+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118539265 0 0) 0x7faa000cca50 con 0x7faa000cd9b0 2024-03-29T11:51:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.695+0000 7fa9f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.695+0000 7fa9f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.695+0000 7fa9f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.695+0000 7fa9f6ffd640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:51:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.695+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e8004eb0 con 0x7faa000cd9b0 2024-03-29T11:51:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411607068 0 0) 0x7faa000ca290 con 0x7faa00006a70 2024-03-29T11:51:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:47.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa000cca50 con 0x7faa00006a70 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2734392142 0 0) 0x7fa9e8004eb0 con 0x7faa000cd9b0 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa000ca290 con 0x7faa000cd9b0 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7faa0804fdd0 con 0x7faa000cd9b0 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 10 monclient(hunting): dump: 2024-03-29T11:51:47.704 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.696+0000 7fa9f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.697+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1582470555 0 0) 0x7faa000cca50 con 0x7faa00006a70 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.697+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e8004eb0 con 0x7faa00006a70 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.697+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa9f8001930 con 0x7faa00006a70 2024-03-29T11:51:47.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.697+0000 7fa9f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.697+0000 7fa9f6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.697+0000 7fa9f6ffd640 10 monclient(hunting): dump: 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:47.706 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:47.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.698+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3698499707 0 0) 0x7fa9e8004eb0 con 0x7faa00006a70 2024-03-29T11:51:47.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.698+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 >> v1:172.21.15.55:6790/0 conn(0x7faa00029c30 legacy=0x7faa000ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:47.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.698+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 >> v1:172.21.15.171:6789/0 conn(0x7faa000cd9b0 legacy=0x7faa000cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:47.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.698+0000 7fa9f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:51:47.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.698+0000 7fa9f6ffd640 1 monclient: found mon.a 2024-03-29T11:51:47.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.698+0000 7fa9f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.698+0000 7fa9f6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.698+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa000cdf90 con 0x7faa00006a70 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.698+0000 7fa9f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7fa9f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7fa9f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9f80047e0 con 0x7faa00006a70 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7fa9f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fa9f8004b70 con 0x7faa00006a70 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7fa9f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7fa9f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7fa9f6ffd640 10 monclient: dump: 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:47.710 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7faa0759e640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7faa0759e640 10 monclient: _renew_subs 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7faa0759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.699+0000 7faa0759e640 1 -- 172.21.15.55:0/43386796 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa000ccf70 con 0x7faa00006a70 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.700+0000 7faa0759e640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.700+0000 7faa0759e640 10 monclient: _renew_subs 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.700+0000 7faa0759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:47.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.700+0000 7faa0759e640 1 -- 172.21.15.55:0/43386796 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa000bfd60 con 0x7faa00006a70 2024-03-29T11:51:47.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.700+0000 7faa0759e640 1 librados: init done 2024-03-29T11:51:47.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.700+0000 7fa9f7fff640 4 set_mon_vals no callback set 2024-03-29T11:51:47.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.701+0000 7fa9f4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:47.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.701+0000 7fa9f4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:47.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.701+0000 7fa9f4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:47.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.701+0000 7fa9f4ff9640 1 -- 172.21.15.55:0/43386796 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa000ca060 con 0x7faa00006a70 2024-03-29T11:51:47.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.702+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fa9f8004cd0 con 0x7faa00006a70 2024-03-29T11:51:47.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.703+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fa9f8084b60 con 0x7faa00006a70 2024-03-29T11:51:47.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.703+0000 7fa9f6ffd640 10 client.4584.objecter ms_dispatch 0x7faa08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:47.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.703+0000 7fa9f6ffd640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:47.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.703+0000 7fa9f6ffd640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:47.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.703+0000 7fa9f6ffd640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-29T11:51:47.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.703+0000 7faa05d9b640 10 monclient: get_auth_request con 0x7fa9e806dce0 auth_method 0 2024-03-29T11:51:47.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.706+0000 7fa9f6ffd640 10 client.4584.objecter ms_handle_connect 0x7fa9e806dce0 2024-03-29T11:51:47.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.711+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa000ca060 con 0x7faa00006a70 2024-03-29T11:51:47.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.711+0000 7fa9f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:47.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:47.711+0000 7fa9f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:48.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.050+0000 7fa9f4ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:48.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.050+0000 7fa9f4ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:48.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.050+0000 7fa9f4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:51:48.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.050+0000 7fa9f4ff9640 1 -- 172.21.15.55:0/43386796 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa000ca060 con 0x7faa00006a70 2024-03-29T11:51:48.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.052+0000 7fa9f6ffd640 1 -- 172.21.15.55:0/43386796 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7faa000ca060 con 0x7faa00006a70 2024-03-29T11:51:48.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.052+0000 7fa9f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:48.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.052+0000 7fa9f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:48.061 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:51:48.061 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:48.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.057+0000 7fa9f4ff9640 10 librados: watch_flush enter 2024-03-29T11:51:48.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.057+0000 7fa9f4ff9640 10 librados: watch_flush exit 2024-03-29T11:51:48.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.057+0000 7fa9f4ff9640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-29T11:51:48.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.057+0000 7fa9f4ff9640 10 client.4584.objecter successfully canceled tick 2024-03-29T11:51:48.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.058+0000 7fa9f4ff9640 1 -- 172.21.15.55:0/43386796 >> v1:172.21.15.171:6816/1463197163 conn(0x7fa9e806dce0 legacy=0x7fa9e80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:48.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.058+0000 7fa9f4ff9640 10 monclient: shutdown 2024-03-29T11:51:48.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.058+0000 7fa9f4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:48.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.058+0000 7fa9f4ff9640 1 -- 172.21.15.55:0/43386796 >> v1:172.21.15.55:6789/0 conn(0x7faa00006a70 legacy=0x7faa000bf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:48.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.058+0000 7fa9f4ff9640 1 -- 172.21.15.55:0/43386796 shutdown_connections 2024-03-29T11:51:48.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.058+0000 7fa9f4ff9640 1 -- 172.21.15.55:0/43386796 >> 172.21.15.55:0/43386796 conn(0x7faa000058f0 msgr2=0x7faa00036a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:48.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.059+0000 7fa9f4ff9640 1 -- 172.21.15.55:0/43386796 shutdown_connections 2024-03-29T11:51:48.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.059+0000 7fa9f4ff9640 1 -- 172.21.15.55:0/43386796 wait complete. 2024-03-29T11:51:48.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:48.059+0000 7fa9f4ff9640 1 librados: shutdown 2024-03-29T11:51:48.082 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:54.889 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.039+0000 7f1740822640 10 monclient: get_monmap_and_config 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.039+0000 7f1740822640 10 monclient: build_initial_monmap 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.039+0000 7f1740822640 10 monclient: monmap: 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:51:55.040907+0000 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:51:55.040907+0000 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:55.047 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:55.048 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:51:55.048 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:51:55.048 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:55.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.040+0000 7f1740822640 1 Processor -- start 2024-03-29T11:51:55.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 1 -- start start 2024-03-29T11:51:55.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 10 monclient: init 2024-03-29T11:51:55.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:55.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:51:55.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 10 monclient(hunting): picked mon.noname-a con 0x7f1734006890 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:55.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 10 monclient(hunting): picked mon.noname-b con 0x7f1734006c70 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:55.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 10 monclient(hunting): picked mon.noname-c con 0x7f173402b4c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:55.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173402c830 con 0x7f1734006890 2024-03-29T11:51:55.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173402afa0 con 0x7f173402b4c0 2024-03-29T11:51:55.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173402aba0 con 0x7f1734006c70 2024-03-29T11:51:55.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:55.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.041+0000 7f1740822640 10 monclient(hunting): authenticate will time out at 1528.321599s 2024-03-29T11:51:55.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.044+0000 7f173bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f173402b4c0 0x7f17340cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:37340/0 (socket says 172.21.15.55:37340) 2024-03-29T11:51:55.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.044+0000 7f173bfff640 1 -- 172.21.15.55:0/2613293442 learned_addr learned my addr 172.21.15.55:0/2613293442 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.045+0000 7f173a7fc640 1 -- 172.21.15.55:0/2613293442 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879125817 0 0) 0x7f173402c830 con 0x7f1734006890 2024-03-29T11:51:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.045+0000 7f173a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:51:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.045+0000 7f173a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.045+0000 7f173a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.045+0000 7f173a7fc640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:51:55.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.045+0000 7f173a7fc640 1 -- 172.21.15.55:0/2613293442 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f172c003520 con 0x7f1734006890 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 1 -- 172.21.15.55:0/2613293442 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1229428188 0 0) 0x7f172c003520 con 0x7f1734006890 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 1 -- 172.21.15.55:0/2613293442 >> v1:172.21.15.55:6790/0 conn(0x7f1734006c70 legacy=0x7f17340c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 1 -- 172.21.15.55:0/2613293442 >> v1:172.21.15.171:6789/0 conn(0x7f173402b4c0 legacy=0x7f17340cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 1 -- 172.21.15.55:0/2613293442 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173402ad30 con 0x7f1734006890 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:51:55.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 1 -- 172.21.15.55:0/2613293442 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f173c04f730 con 0x7f1734006890 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: dump: 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 1 -- 172.21.15.55:0/2613293442 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f173c0008d0 con 0x7f1734006890 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:55.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 1 -- 172.21.15.55:0/2613293442 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f173c0758b0 con 0x7f1734006890 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.047+0000 7f173a7fc640 10 monclient: dump: 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.048+0000 7f1740822640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.048+0000 7f1740822640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:55.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.048+0000 7f1740822640 4 set_mon_vals no callback set 2024-03-29T11:51:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.050+0000 7f1740822640 10 monclient: shutdown 2024-03-29T11:51:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.050+0000 7f1740822640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.050+0000 7f1740822640 1 -- 172.21.15.55:0/2613293442 >> v1:172.21.15.55:6789/0 conn(0x7f1734006890 legacy=0x7f17340c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.050+0000 7f1740822640 1 -- 172.21.15.55:0/2613293442 shutdown_connections 2024-03-29T11:51:55.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.050+0000 7f1740822640 1 -- 172.21.15.55:0/2613293442 >> 172.21.15.55:0/2613293442 conn(0x7f17340058f0 msgr2=0x7f1734005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:55.059 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.051+0000 7f1740822640 1 -- 172.21.15.55:0/2613293442 shutdown_connections 2024-03-29T11:51:55.060 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.051+0000 7f1740822640 1 -- 172.21.15.55:0/2613293442 wait complete. 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.052+0000 7f1740822640 10 monclient: build_initial_monmap 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.052+0000 7f1740822640 10 monclient: monmap: 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:51:55.064 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.053+0000 7f1740822640 1 librados: starting msgr at 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.053+0000 7f1740822640 1 librados: starting objecter 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.053+0000 7f1740822640 1 Processor -- start 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.053+0000 7f1740822640 1 -- start start 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.053+0000 7f1740822640 1 librados: setting wanted keys 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.053+0000 7f1740822640 1 librados: calling monclient init 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.053+0000 7f1740822640 10 monclient: init 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.054+0000 7f1740822640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.054+0000 7f1740822640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.054+0000 7f1740822640 10 monclient(hunting): picked mon.noname-b con 0x7f1734006890 addr v1:172.21.15.171:6789/0 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.054+0000 7f1740822640 10 monclient(hunting): picked mon.noname-c con 0x7f173402b4c0 addr v1:172.21.15.55:6790/0 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.055+0000 7f173b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f1734006890 0x7f17340c3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:40920/0 (socket says 172.21.15.55:40920) 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.057+0000 7f1740822640 10 monclient(hunting): picked mon.noname-a con 0x7f17340c2560 addr v1:172.21.15.55:6789/0 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.057+0000 7f1740822640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17340c4600 con 0x7f17340c2560 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.057+0000 7f1740822640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17340c4320 con 0x7f1734006890 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.057+0000 7f1740822640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173402d8c0 con 0x7f173402b4c0 2024-03-29T11:51:55.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.057+0000 7f1740822640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:55.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.057+0000 7f1740822640 10 monclient(hunting): authenticate will time out at 1528.336777s 2024-03-29T11:51:55.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.057+0000 7f173b7fe640 1 -- 172.21.15.55:0/1549123277 learned_addr learned my addr 172.21.15.55:0/1549123277 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 10 client.?.objecter ms_handle_connect 0x7f173402b4c0 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 10 client.?.objecter ms_handle_connect 0x7f1734006890 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839754508 0 0) 0x7f173402d8c0 con 0x7f173402b4c0 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 10 monclient(hunting): _negotiate 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.058+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1720004eb0 con 0x7f173402b4c0 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182994853 0 0) 0x7f17340c4320 con 0x7f1734006890 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 10 monclient(hunting): _negotiate 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:55.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f173402d8c0 con 0x7f1734006890 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4219581120 0 0) 0x7f1720004eb0 con 0x7f173402b4c0 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17340c4320 con 0x7f173402b4c0 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f1728001930 con 0x7f173402b4c0 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 10 monclient(hunting): dump: 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:55.068 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.059+0000 7f1727fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1126869913 0 0) 0x7f173402d8c0 con 0x7f1734006890 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1720004eb0 con 0x7f1734006890 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f173c04f400 con 0x7f1734006890 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient(hunting): dump: 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:55.069 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:55.070 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:55.070 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:55.070 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:55.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 client.?.objecter ms_handle_connect 0x7f17340c2560 2024-03-29T11:51:55.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:55.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800845695 0 0) 0x7f17340c4600 con 0x7f17340c2560 2024-03-29T11:51:55.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient(hunting): _negotiate 2024-03-29T11:51:55.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f173402d8c0 con 0x7f17340c2560 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3505400260 0 0) 0x7f17340c4320 con 0x7f173402b4c0 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 >> v1:172.21.15.171:6789/0 conn(0x7f1734006890 legacy=0x7f17340c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 >> v1:172.21.15.55:6789/0 conn(0x7f17340c2560 legacy=0x7f173416b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient: _finish_hunting 0 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 monclient: found mon.c 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17340c3a00 con 0x7f173402b4c0 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient: _finish_auth 0 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 10 monclient: _check_auth_tickets 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1727fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1740822640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:51:55.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1740822640 10 monclient: _renew_subs 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1740822640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17340bf610 con 0x7f173402b4c0 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1740822640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1740822640 10 monclient: _renew_subs 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1740822640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f173416d060 con 0x7f173402b4c0 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.060+0000 7f1740822640 1 librados: init done 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1728001a90 con 0x7f173402b4c0 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f17280038d0 con 0x7f173402b4c0 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 10 monclient: dump: 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:51:55.073 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f1728003af0 con 0x7f173402b4c0 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f173b7fe640 10 monclient: get_auth_request con 0x7f172006de10 auth_method 0 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f1728083a10 con 0x7f173402b4c0 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 10 client.4604.objecter ms_dispatch 0x7f173c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.063+0000 7f1727fff640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:51:55.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.064+0000 7f1727fff640 10 client.4604.objecter ms_handle_connect 0x7f172006de10 2024-03-29T11:51:55.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.064+0000 7f1740822640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:55.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.064+0000 7f1740822640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:55.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.064+0000 7f1740822640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:55.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.064+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17340c0fc0 con 0x7f173402b4c0 2024-03-29T11:51:55.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.064+0000 7f1739ffb640 4 set_mon_vals no callback set 2024-03-29T11:51:55.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.072+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f172804f370 con 0x7f173402b4c0 2024-03-29T11:51:55.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.072+0000 7f1727fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:55.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.072+0000 7f1727fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:55.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.404+0000 7f1740822640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:55.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.404+0000 7f1740822640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:55.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.404+0000 7f1740822640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:51:55.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.404+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f173402a520 con 0x7f173402b4c0 2024-03-29T11:51:55.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.405+0000 7f1727fff640 1 -- 172.21.15.55:0/1549123277 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f17280512a0 con 0x7f173402b4c0 2024-03-29T11:51:55.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.405+0000 7f1727fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:55.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.405+0000 7f1727fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:55.414 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:51:55.414 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:55.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.408+0000 7f1740822640 10 librados: watch_flush enter 2024-03-29T11:51:55.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.408+0000 7f1740822640 10 librados: watch_flush exit 2024-03-29T11:51:55.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.408+0000 7f1740822640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:51:55.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.408+0000 7f1740822640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:51:55.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.408+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 >> v1:172.21.15.171:6816/1463197163 conn(0x7f172006de10 legacy=0x7f17200702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:55.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.408+0000 7f1740822640 10 monclient: shutdown 2024-03-29T11:51:55.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.408+0000 7f1740822640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:55.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.408+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 >> v1:172.21.15.55:6790/0 conn(0x7f173402b4c0 legacy=0x7f173402d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:55.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.409+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 shutdown_connections 2024-03-29T11:51:55.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.409+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 >> 172.21.15.55:0/1549123277 conn(0x7f17340058f0 msgr2=0x7f1734038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:55.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.409+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 shutdown_connections 2024-03-29T11:51:55.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.409+0000 7f1740822640 1 -- 172.21.15.55:0/1549123277 wait complete. 2024-03-29T11:51:55.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:51:55.409+0000 7f1740822640 1 librados: shutdown 2024-03-29T11:51:55.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:02.239 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.365+0000 7f0625732640 10 monclient: get_monmap_and_config 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.365+0000 7f0625732640 10 monclient: build_initial_monmap 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.365+0000 7f0625732640 10 monclient: monmap: 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:52:02.366897+0000 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:52:02.366897+0000 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:02.373 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:52:02.374 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:52:02.374 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:02.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.366+0000 7f0625732640 1 Processor -- start 2024-03-29T11:52:02.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.366+0000 7f0625732640 1 -- start start 2024-03-29T11:52:02.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.366+0000 7f0625732640 10 monclient: init 2024-03-29T11:52:02.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.367+0000 7f0625732640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:02.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.367+0000 7f0625732640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:52:02.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.367+0000 7f0625732640 10 monclient(hunting): picked mon.noname-c con 0x7f0618038a10 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:02.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.367+0000 7f0625732640 10 monclient(hunting): picked mon.noname-b con 0x7f0618038df0 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:02.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.367+0000 7f061e7fc640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f0618038df0 0x7f06180cae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:60542/0 (socket says 172.21.15.55:60542) 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f061e7fc640 1 -- 172.21.15.55:0/3101872747 learned_addr learned my addr 172.21.15.55:0/3101872747 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f0625732640 10 monclient(hunting): picked mon.noname-a con 0x7f0618030690 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f0625732640 1 -- 172.21.15.55:0/3101872747 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180c54f0 con 0x7f0618030690 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f0625732640 1 -- 172.21.15.55:0/3101872747 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0618036460 con 0x7f0618038a10 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f0625732640 1 -- 172.21.15.55:0/3101872747 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0618039870 con 0x7f0618038df0 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f0625732640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f0625732640 10 monclient(hunting): authenticate will time out at 1535.648130s 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f061dffb640 1 -- 172.21.15.55:0/3101872747 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292171491 0 0) 0x7f0618039870 con 0x7f0618038df0 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f061dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:02.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f061dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:02.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f061dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:02.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f061dffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:52:02.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.368+0000 7f061dffb640 1 -- 172.21.15.55:0/3101872747 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0608003520 con 0x7f0618038df0 2024-03-29T11:52:02.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.369+0000 7f061dffb640 1 -- 172.21.15.55:0/3101872747 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063167569 0 0) 0x7f0608003520 con 0x7f0618038df0 2024-03-29T11:52:02.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.369+0000 7f061dffb640 1 -- 172.21.15.55:0/3101872747 >> v1:172.21.15.171:6789/0 conn(0x7f0618038a10 legacy=0x7f0618033550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:52:02.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.369+0000 7f061dffb640 1 -- 172.21.15.55:0/3101872747 >> v1:172.21.15.55:6789/0 conn(0x7f0618030690 legacy=0x7f06180d31a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:52:02.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:52:02.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 1 monclient: found mon.noname-b 2024-03-29T11:52:02.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 1 -- 172.21.15.55:0/3101872747 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06180356c0 con 0x7f0618038df0 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: _finish_auth 0 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 1 -- 172.21.15.55:0/3101872747 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0614001690 con 0x7f0618038df0 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: dump: 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:02.385 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f0625732640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f0625732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 1 -- 172.21.15.55:0/3101872747 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0614001860 con 0x7f0618038df0 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:02.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 1 -- 172.21.15.55:0/3101872747 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0614001cb0 con 0x7f0618038df0 2024-03-29T11:52:02.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:02.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:02.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f061dffb640 10 monclient: dump: 2024-03-29T11:52:02.387 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:02.387 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:02.387 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.387 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.387 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:02.387 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f0625732640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.371+0000 7f0625732640 4 set_mon_vals no callback set 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.372+0000 7f0625732640 10 monclient: shutdown 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.372+0000 7f0625732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.372+0000 7f0625732640 1 -- 172.21.15.55:0/3101872747 >> v1:172.21.15.55:6790/0 conn(0x7f0618038df0 legacy=0x7f06180cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.372+0000 7f0625732640 1 -- 172.21.15.55:0/3101872747 shutdown_connections 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.372+0000 7f0625732640 1 -- 172.21.15.55:0/3101872747 >> 172.21.15.55:0/3101872747 conn(0x7f0618005c30 msgr2=0x7f0618006040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.372+0000 7f0625732640 1 -- 172.21.15.55:0/3101872747 shutdown_connections 2024-03-29T11:52:02.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.373+0000 7f0625732640 1 -- 172.21.15.55:0/3101872747 wait complete. 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.374+0000 7f0625732640 10 monclient: build_initial_monmap 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.374+0000 7f0625732640 10 monclient: monmap: 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:52:02.389 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.374+0000 7f0625732640 1 librados: starting msgr at 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.374+0000 7f0625732640 1 librados: starting objecter 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.375+0000 7f0625732640 1 Processor -- start 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.375+0000 7f0625732640 1 -- start start 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.375+0000 7f0625732640 1 librados: setting wanted keys 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.375+0000 7f0625732640 1 librados: calling monclient init 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.375+0000 7f0625732640 10 monclient: init 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.375+0000 7f0625732640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.375+0000 7f0625732640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f0625732640 10 monclient(hunting): picked mon.noname-b con 0x7f0618030690 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f0625732640 10 monclient(hunting): picked mon.noname-c con 0x7f0618038a10 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f0625732640 10 monclient(hunting): picked mon.noname-a con 0x7f061802fba0 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:02.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f0625732640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180c6ba0 con 0x7f061802fba0 2024-03-29T11:52:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f0625732640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180c6e10 con 0x7f0618030690 2024-03-29T11:52:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f0625732640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180c5ef0 con 0x7f0618038a10 2024-03-29T11:52:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f0625732640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f0625732640 10 monclient(hunting): authenticate will time out at 1535.655872s 2024-03-29T11:52:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f061effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0618030690 0x7f06180c79d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:35222/0 (socket says 172.21.15.55:35222) 2024-03-29T11:52:02.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.376+0000 7f061effd640 1 -- 172.21.15.55:0/3922278557 learned_addr learned my addr 172.21.15.55:0/3922278557 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 client.?.objecter ms_handle_connect 0x7f0618030690 2024-03-29T11:52:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 client.?.objecter ms_handle_connect 0x7f0618038a10 2024-03-29T11:52:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117100776 0 0) 0x7f06180c6e10 con 0x7f0618030690 2024-03-29T11:52:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:52:02.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f060c004eb0 con 0x7f0618030690 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169668589 0 0) 0x7f06180c5ef0 con 0x7f0618038a10 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06180c6e10 con 0x7f0618038a10 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3861672190 0 0) 0x7f060c004eb0 con 0x7f0618030690 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06180c5ef0 con 0x7f0618030690 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0610001960 con 0x7f0618030690 2024-03-29T11:52:02.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.378+0000 7f06077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 10 monclient(hunting): dump: 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:02.398 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3835022048 0 0) 0x7f06180c6e10 con 0x7f0618038a10 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f060c004eb0 con 0x7f0618038a10 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0614001b70 con 0x7f0618038a10 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 10 monclient(hunting): dump: 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.399 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3790612230 0 0) 0x7f06180c5ef0 con 0x7f0618030690 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 >> v1:172.21.15.55:6790/0 conn(0x7f0618038a10 legacy=0x7f06180c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.379+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 >> v1:172.21.15.55:6789/0 conn(0x7f061802fba0 legacy=0x7f06181671c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:52:02.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 1 monclient: found mon.b 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06180c6080 con 0x7f0618030690 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 10 monclient: _finish_auth 0 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f0625732640 5 monclient: authenticate success, global_id 4561 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f0625732640 10 monclient: _renew_subs 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f0625732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06180c58f0 con 0x7f0618030690 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f0625732640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f0625732640 10 monclient: _renew_subs 2024-03-29T11:52:02.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f0625732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06181688e0 con 0x7f0618030690 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f0625732640 1 librados: init done 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06100035f0 con 0x7f0618030690 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f061d7fa640 4 set_mon_vals no callback set 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0610003940 con 0x7f0618030690 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.380+0000 7f06077fe640 10 monclient: dump: 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.402 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.383+0000 7f0625732640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.383+0000 7f0625732640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.383+0000 7f0625732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.383+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0600005080 con 0x7f0618030690 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.384+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f0610003aa0 con 0x7f0618030690 2024-03-29T11:52:02.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.384+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7f0610084a20 con 0x7f0618030690 2024-03-29T11:52:02.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.384+0000 7f061e7fc640 10 monclient: get_auth_request con 0x7f060c06ded0 auth_method 0 2024-03-29T11:52:02.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.384+0000 7f06077fe640 10 client.4561.objecter ms_dispatch 0x7f0620000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:02.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.384+0000 7f06077fe640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:02.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.384+0000 7f06077fe640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:02.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.384+0000 7f06077fe640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-29T11:52:02.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.385+0000 7f06077fe640 10 client.4561.objecter ms_handle_connect 0x7f060c06ded0 2024-03-29T11:52:02.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.393+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0610050300 con 0x7f0618030690 2024-03-29T11:52:02.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.393+0000 7f06077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:02.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.393+0000 7f06077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:02.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.715+0000 7f0625732640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:02.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.715+0000 7f0625732640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:02.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.715+0000 7f0625732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:02.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.715+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0600005330 con 0x7f0618030690 2024-03-29T11:52:02.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.718+0000 7f06077fe640 1 -- 172.21.15.55:0/3922278557 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7f0610052230 con 0x7f0618030690 2024-03-29T11:52:02.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.718+0000 7f06077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:02.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.718+0000 7f06077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:02.726 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:52:02.726 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:02.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.721+0000 7f0625732640 10 librados: watch_flush enter 2024-03-29T11:52:02.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.721+0000 7f0625732640 10 librados: watch_flush exit 2024-03-29T11:52:02.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.721+0000 7f0625732640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-29T11:52:02.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.721+0000 7f0625732640 10 client.4561.objecter successfully canceled tick 2024-03-29T11:52:02.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.723+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 >> v1:172.21.15.171:6816/1463197163 conn(0x7f060c06ded0 legacy=0x7f060c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:02.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.723+0000 7f0625732640 10 monclient: shutdown 2024-03-29T11:52:02.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.723+0000 7f0625732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:02.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.723+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 >> v1:172.21.15.171:6789/0 conn(0x7f0618030690 legacy=0x7f06180c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:02.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.724+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 shutdown_connections 2024-03-29T11:52:02.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.724+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 >> 172.21.15.55:0/3922278557 conn(0x7f0618005c30 msgr2=0x7f06180c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:02.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.724+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 shutdown_connections 2024-03-29T11:52:02.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.724+0000 7f0625732640 1 -- 172.21.15.55:0/3922278557 wait complete. 2024-03-29T11:52:02.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:02.724+0000 7f0625732640 1 librados: shutdown 2024-03-29T11:52:02.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:09.556 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:09.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.682+0000 7fab4ef10640 10 monclient: get_monmap_and_config 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.682+0000 7fab4ef10640 10 monclient: build_initial_monmap 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.682+0000 7fab4ef10640 10 monclient: monmap: 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:52:09.683607+0000 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:52:09.683607+0000 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:52:09.690 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:09.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.683+0000 7fab4ef10640 1 Processor -- start 2024-03-29T11:52:09.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.683+0000 7fab4ef10640 1 -- start start 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.683+0000 7fab4ef10640 10 monclient: init 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 10 monclient(hunting): picked mon.noname-a con 0x7fab480a9750 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 10 monclient(hunting): picked mon.noname-b con 0x7fab4813e1e0 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 10 monclient(hunting): picked mon.noname-c con 0x7fab48071c30 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480ab300 con 0x7fab480a9750 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480ac2f0 con 0x7fab48071c30 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480ad8d0 con 0x7fab4813e1e0 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4d4ad640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fab48071c30 0x7fab480ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:32906/0 (socket says 172.21.15.55:32906) 2024-03-29T11:52:09.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4d4ad640 1 -- 172.21.15.55:0/1785506350 learned_addr learned my addr 172.21.15.55:0/1785506350 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.684+0000 7fab4ef10640 10 monclient(hunting): authenticate will time out at 1542.964392s 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.685+0000 7fab477fe640 1 -- 172.21.15.55:0/1785506350 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131034030 0 0) 0x7fab480ad8d0 con 0x7fab4813e1e0 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.685+0000 7fab477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.685+0000 7fab477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.685+0000 7fab477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.685+0000 7fab477fe640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.685+0000 7fab477fe640 1 -- 172.21.15.55:0/1785506350 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab30003520 con 0x7fab4813e1e0 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.685+0000 7fab477fe640 1 -- 172.21.15.55:0/1785506350 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3734425542 0 0) 0x7fab30003520 con 0x7fab4813e1e0 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.685+0000 7fab477fe640 1 -- 172.21.15.55:0/1785506350 >> v1:172.21.15.171:6789/0 conn(0x7fab48071c30 legacy=0x7fab480ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:09.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.685+0000 7fab477fe640 1 -- 172.21.15.55:0/1785506350 >> v1:172.21.15.55:6789/0 conn(0x7fab480a9750 legacy=0x7fab4813dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 1 monclient: found mon.noname-b 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 1 -- 172.21.15.55:0/1785506350 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab480ada60 con 0x7fab4813e1e0 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 10 monclient: _finish_auth 0 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 1 -- 172.21.15.55:0/1785506350 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fab3c0015c0 con 0x7fab4813e1e0 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:09.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab477fe640 10 monclient: dump: 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab4ef10640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.687+0000 7fab4ef10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.688+0000 7fab477fe640 1 -- 172.21.15.55:0/1785506350 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab3c001790 con 0x7fab4813e1e0 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.688+0000 7fab477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.688+0000 7fab4ef10640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:09.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.688+0000 7fab4ef10640 4 set_mon_vals no callback set 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.688+0000 7fab477fe640 1 -- 172.21.15.55:0/1785506350 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fab3c001be0 con 0x7fab4813e1e0 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.688+0000 7fab477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.688+0000 7fab477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.688+0000 7fab477fe640 10 monclient: dump: 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:09.698 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:09.699 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:09.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.689+0000 7fab4ef10640 10 monclient: shutdown 2024-03-29T11:52:09.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.689+0000 7fab4ef10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:09.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.689+0000 7fab4ef10640 1 -- 172.21.15.55:0/1785506350 >> v1:172.21.15.55:6790/0 conn(0x7fab4813e1e0 legacy=0x7fab480ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:09.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.689+0000 7fab4ef10640 1 -- 172.21.15.55:0/1785506350 shutdown_connections 2024-03-29T11:52:09.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.689+0000 7fab4ef10640 1 -- 172.21.15.55:0/1785506350 >> 172.21.15.55:0/1785506350 conn(0x7fab4806b660 msgr2=0x7fab4806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:09.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.690+0000 7fab4ef10640 1 -- 172.21.15.55:0/1785506350 shutdown_connections 2024-03-29T11:52:09.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.690+0000 7fab4ef10640 1 -- 172.21.15.55:0/1785506350 wait complete. 2024-03-29T11:52:09.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.691+0000 7fab4ef10640 10 monclient: build_initial_monmap 2024-03-29T11:52:09.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.691+0000 7fab4ef10640 10 monclient: monmap: 2024-03-29T11:52:09.700 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:09.700 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:09.700 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.691+0000 7fab4ef10640 1 librados: starting msgr at 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.691+0000 7fab4ef10640 1 librados: starting objecter 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.692+0000 7fab4ef10640 1 Processor -- start 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.692+0000 7fab4ef10640 1 -- start start 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.692+0000 7fab4ef10640 1 librados: setting wanted keys 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.692+0000 7fab4ef10640 1 librados: calling monclient init 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.692+0000 7fab4ef10640 10 monclient: init 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.693+0000 7fab4ef10640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:09.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.693+0000 7fab4ef10640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:52:09.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.693+0000 7fab4ef10640 10 monclient(hunting): picked mon.noname-b con 0x7fab48071c30 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:09.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab4ef10640 10 monclient(hunting): picked mon.noname-a con 0x7fab480a9750 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:09.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab4ef10640 10 monclient(hunting): picked mon.noname-c con 0x7fab4813e1e0 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:09.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab4ef10640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480b2f60 con 0x7fab480a9750 2024-03-29T11:52:09.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab4ef10640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab481cc970 con 0x7fab48071c30 2024-03-29T11:52:09.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab4ef10640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab48139780 con 0x7fab4813e1e0 2024-03-29T11:52:09.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab4ef10640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:09.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab4d4ad640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fab4813e1e0 0x7fab481e7020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:37856/0 (socket says 172.21.15.55:37856) 2024-03-29T11:52:09.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab4d4ad640 1 -- 172.21.15.55:0/3459480419 learned_addr learned my addr 172.21.15.55:0/3459480419 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab4ef10640 10 monclient(hunting): authenticate will time out at 1542.974147s 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab457fa640 10 client.?.objecter ms_handle_connect 0x7fab4813e1e0 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800643373 0 0) 0x7fab48139780 con 0x7fab4813e1e0 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab457fa640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.694+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34004eb0 con 0x7fab4813e1e0 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 client.?.objecter ms_handle_connect 0x7fab48071c30 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 955955799 0 0) 0x7fab34004eb0 con 0x7fab4813e1e0 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab48139780 con 0x7fab4813e1e0 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fab40001c30 con 0x7fab4813e1e0 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:09.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 monclient(hunting): dump: 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659165055 0 0) 0x7fab481cc970 con 0x7fab48071c30 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:09.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:52:09.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34004eb0 con 0x7fab48071c30 2024-03-29T11:52:09.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1744011064 0 0) 0x7fab48139780 con 0x7fab4813e1e0 2024-03-29T11:52:09.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 >> v1:172.21.15.171:6789/0 conn(0x7fab48071c30 legacy=0x7fab48139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:09.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.695+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 >> v1:172.21.15.55:6789/0 conn(0x7fab480a9750 legacy=0x7fab481dece0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:09.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 10 monclient: _finish_hunting 0 2024-03-29T11:52:09.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 1 monclient: found mon.c 2024-03-29T11:52:09.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:09.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab48139910 con 0x7fab4813e1e0 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 10 monclient: _finish_auth 0 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 10 monclient: _check_auth_tickets 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab4ef10640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab4ef10640 10 monclient: _renew_subs 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab4ef10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab480b29d0 con 0x7fab4813e1e0 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab40004b60 con 0x7fab4813e1e0 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fab40004f70 con 0x7fab4813e1e0 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab457fa640 10 monclient: dump: 2024-03-29T11:52:09.709 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.697+0000 7fab46ffd640 4 set_mon_vals no callback set 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.698+0000 7fab4ef10640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.698+0000 7fab4ef10640 10 monclient: _renew_subs 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.698+0000 7fab4ef10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.698+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab481cc600 con 0x7fab4813e1e0 2024-03-29T11:52:09.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.698+0000 7fab4ef10640 1 librados: init done 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.700+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fab400050d0 con 0x7fab4813e1e0 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.700+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 819076827 0 0) 0x7fab40084f10 con 0x7fab4813e1e0 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.700+0000 7fab457fa640 10 client.4616.objecter ms_dispatch 0x7fab48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.700+0000 7fab457fa640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.700+0000 7fab457fa640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.700+0000 7fab457fa640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.701+0000 7fab4ccac640 10 monclient: get_auth_request con 0x7fab3406de20 auth_method 0 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.701+0000 7fab4ef10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.701+0000 7fab4ef10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.701+0000 7fab4ef10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.701+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab480b2d70 con 0x7fab4813e1e0 2024-03-29T11:52:09.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.701+0000 7fab457fa640 10 client.4616.objecter ms_handle_connect 0x7fab3406de20 2024-03-29T11:52:09.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.708+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab400507f0 con 0x7fab4813e1e0 2024-03-29T11:52:09.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.708+0000 7fab457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:09.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:09.708+0000 7fab457fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:10.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.030+0000 7fab4ef10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:10.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.030+0000 7fab4ef10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:10.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.030+0000 7fab4ef10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:10.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.030+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab48075f10 con 0x7fab4813e1e0 2024-03-29T11:52:10.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.035+0000 7fab457fa640 1 -- 172.21.15.55:0/3459480419 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274864823) 0x7fab40052720 con 0x7fab4813e1e0 2024-03-29T11:52:10.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.035+0000 7fab457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:10.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.035+0000 7fab457fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:10.043 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:52:10.043 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:42:12.285214+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:12.285214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:10.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.038+0000 7fab4ef10640 10 librados: watch_flush enter 2024-03-29T11:52:10.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.038+0000 7fab4ef10640 10 librados: watch_flush exit 2024-03-29T11:52:10.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.038+0000 7fab4ef10640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-29T11:52:10.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.038+0000 7fab4ef10640 10 client.4616.objecter successfully canceled tick 2024-03-29T11:52:10.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.038+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 >> v1:172.21.15.171:6816/1463197163 conn(0x7fab3406de20 legacy=0x7fab340702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:10.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.038+0000 7fab4ef10640 10 monclient: shutdown 2024-03-29T11:52:10.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.038+0000 7fab4ef10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:10.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.038+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 >> v1:172.21.15.55:6790/0 conn(0x7fab4813e1e0 legacy=0x7fab481e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:10.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.039+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 shutdown_connections 2024-03-29T11:52:10.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.039+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 >> 172.21.15.55:0/3459480419 conn(0x7fab4806b660 msgr2=0x7fab4813c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:10.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.039+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 shutdown_connections 2024-03-29T11:52:10.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.039+0000 7fab4ef10640 1 -- 172.21.15.55:0/3459480419 wait complete. 2024-03-29T11:52:10.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:10.039+0000 7fab4ef10640 1 librados: shutdown 2024-03-29T11:52:10.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:16.871 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:17.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.022+0000 7fb20192b640 10 monclient: get_monmap_and_config 2024-03-29T11:52:17.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.022+0000 7fb20192b640 10 monclient: build_initial_monmap 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.022+0000 7fb20192b640 10 monclient: monmap: 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:52:17.022753+0000 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:52:17.022753+0000 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:52:17.030 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:17.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.025+0000 7fb20192b640 1 Processor -- start 2024-03-29T11:52:17.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.025+0000 7fb20192b640 1 -- start start 2024-03-29T11:52:17.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.025+0000 7fb20192b640 10 monclient: init 2024-03-29T11:52:17.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:17.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:52:17.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 10 monclient(hunting): picked mon.noname-c con 0x7fb1fc0a9ac0 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:17.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 10 monclient(hunting): picked mon.noname-a con 0x7fb1fc13ee90 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:17.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 10 monclient(hunting): picked mon.noname-b con 0x7fb1fc0af900 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:17.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc0afd10 con 0x7fb1fc13ee90 2024-03-29T11:52:17.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc0af6f0 con 0x7fb1fc0a9ac0 2024-03-29T11:52:17.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc0aac80 con 0x7fb1fc0af900 2024-03-29T11:52:17.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:17.033 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.026+0000 7fb20192b640 10 monclient(hunting): authenticate will time out at 1550.305130s 2024-03-29T11:52:17.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.030+0000 7fb1fb7fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fb1fc0af900 0x7fb1fc149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:35471/0 (socket says 172.21.15.55:35471) 2024-03-29T11:52:17.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.030+0000 7fb1fb7fe640 1 -- 172.21.15.55:0/3808978522 learned_addr learned my addr 172.21.15.55:0/3808978522 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.033+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714961140 0 0) 0x7fb1fc0af6f0 con 0x7fb1fc0a9ac0 2024-03-29T11:52:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e4003520 con 0x7fb1fc0a9ac0 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918186989 0 0) 0x7fb1fc0aac80 con 0x7fb1fc0af900 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1fc0af6f0 con 0x7fb1fc0af900 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829874601 0 0) 0x7fb1fc0afd10 con 0x7fb1fc13ee90 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1fc0aac80 con 0x7fb1fc13ee90 2024-03-29T11:52:17.041 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 182974701 0 0) 0x7fb1fc0af6f0 con 0x7fb1fc0af900 2024-03-29T11:52:17.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 >> v1:172.21.15.171:6789/0 conn(0x7fb1fc0a9ac0 legacy=0x7fb1fc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:17.042 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.034+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 >> v1:172.21.15.55:6789/0 conn(0x7fb1fc13ee90 legacy=0x7fb1fc13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:17.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:52:17.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:52:17.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:17.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:52:17.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc0aaf60 con 0x7fb1fc0af900 2024-03-29T11:52:17.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fb1ec0015f0 con 0x7fb1fc0af900 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 10 monclient: dump: 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:17.044 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb20192b640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb20192b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1ec0017d0 con 0x7fb1fc0af900 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.035+0000 7fb1f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.036+0000 7fb20192b640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.036+0000 7fb20192b640 4 set_mon_vals no callback set 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.036+0000 7fb1f9ffb640 1 -- 172.21.15.55:0/3808978522 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fb1ec001be0 con 0x7fb1fc0af900 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.036+0000 7fb1f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.036+0000 7fb1f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.036+0000 7fb1f9ffb640 10 monclient: dump: 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.045 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.036+0000 7fb20192b640 10 monclient: shutdown 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.037+0000 7fb20192b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.037+0000 7fb20192b640 1 -- 172.21.15.55:0/3808978522 >> v1:172.21.15.55:6790/0 conn(0x7fb1fc0af900 legacy=0x7fb1fc149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.037+0000 7fb20192b640 1 -- 172.21.15.55:0/3808978522 shutdown_connections 2024-03-29T11:52:17.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.037+0000 7fb20192b640 1 -- 172.21.15.55:0/3808978522 >> 172.21.15.55:0/3808978522 conn(0x7fb1fc0b35e0 msgr2=0x7fb1fc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:17.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.041+0000 7fb20192b640 1 -- 172.21.15.55:0/3808978522 shutdown_connections 2024-03-29T11:52:17.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.042+0000 7fb20192b640 1 -- 172.21.15.55:0/3808978522 wait complete. 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.043+0000 7fb20192b640 10 monclient: build_initial_monmap 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.043+0000 7fb20192b640 10 monclient: monmap: 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:52:17.051 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.044+0000 7fb20192b640 1 librados: starting msgr at 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.044+0000 7fb20192b640 1 librados: starting objecter 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.044+0000 7fb20192b640 1 Processor -- start 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.045+0000 7fb20192b640 1 -- start start 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.045+0000 7fb20192b640 1 librados: setting wanted keys 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.045+0000 7fb20192b640 1 librados: calling monclient init 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.045+0000 7fb20192b640 10 monclient: init 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.045+0000 7fb20192b640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:17.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.046+0000 7fb20192b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:52:17.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.049+0000 7fb20192b640 10 monclient(hunting): picked mon.noname-c con 0x7fb1fc0af900 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:17.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.049+0000 7fb1faffd640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fb1fc0af900 0x7fb1fc1d62c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:35615/0 (socket says 172.21.15.55:35615) 2024-03-29T11:52:17.056 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.049+0000 7fb1faffd640 1 -- 172.21.15.55:0/2850462787 learned_addr learned my addr 172.21.15.55:0/2850462787 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:17.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.049+0000 7fb1e3fff640 10 client.?.objecter ms_handle_connect 0x7fb1fc0af900 2024-03-29T11:52:17.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.051+0000 7fb1e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:17.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.051+0000 7fb20192b640 10 monclient(hunting): picked mon.noname-a con 0x7fb1fc13ee90 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:17.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.051+0000 7fb20192b640 10 monclient(hunting): picked mon.noname-b con 0x7fb1fc13f5c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:17.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.051+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc140690 con 0x7fb1fc13ee90 2024-03-29T11:52:17.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.051+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc1384d0 con 0x7fb1fc13f5c0 2024-03-29T11:52:17.057 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.051+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc139b90 con 0x7fb1fc0af900 2024-03-29T11:52:17.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.051+0000 7fb20192b640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:17.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.051+0000 7fb20192b640 10 monclient(hunting): authenticate will time out at 1550.329941s 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.054+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726183608 0 0) 0x7fb1fc139b90 con 0x7fb1fc0af900 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e8004eb0 con 0x7fb1fc0af900 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 client.?.objecter ms_handle_connect 0x7fb1fc13ee90 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 client.?.objecter ms_handle_connect 0x7fb1fc13f5c0 2024-03-29T11:52:17.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682772936 0 0) 0x7fb1fc140690 con 0x7fb1fc13ee90 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.055+0000 7fb1e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1fc139b90 con 0x7fb1fc13ee90 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959469311 0 0) 0x7fb1fc1384d0 con 0x7fb1fc13f5c0 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1fc140690 con 0x7fb1fc13f5c0 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 6145817 0 0) 0x7fb1e8004eb0 con 0x7fb1fc0af900 2024-03-29T11:52:17.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1fc1384d0 con 0x7fb1fc0af900 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4199903739 0 0) 0x7fb1fc139b90 con 0x7fb1fc13ee90 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.056+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e8004eb0 con 0x7fb1fc13ee90 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fb1f00018d0 con 0x7fb1fc0af900 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 10 monclient(hunting): dump: 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:17.069 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fb1f4001b50 con 0x7fb1fc13ee90 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 10 monclient(hunting): dump: 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.070 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 950259927 0 0) 0x7fb1fc1384d0 con 0x7fb1fc0af900 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 >> v1:172.21.15.171:6789/0 conn(0x7fb1fc13f5c0 legacy=0x7fb1fc137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 >> v1:172.21.15.55:6789/0 conn(0x7fb1fc13ee90 legacy=0x7fb1fc1376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.057+0000 7fb1e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb1e3fff640 1 monclient: found mon.c 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb1e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb1e3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc139e70 con 0x7fb1fc0af900 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb1e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb1e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:52:17.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb1e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1f0003610 con 0x7fb1fc0af900 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb20192b640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb20192b640 10 monclient: _renew_subs 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb20192b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc1d69d0 con 0x7fb1fc0af900 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb20192b640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb20192b640 10 monclient: _renew_subs 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.058+0000 7fb20192b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.059+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc13a390 con 0x7fb1fc0af900 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.059+0000 7fb20192b640 1 librados: init done 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.059+0000 7fb1e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.059+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fb1f00039e0 con 0x7fb1fc0af900 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.059+0000 7fb1e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.059+0000 7fb1f97fa640 4 set_mon_vals no callback set 2024-03-29T11:52:17.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.059+0000 7fb1e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.060+0000 7fb1e3fff640 10 monclient: dump: 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:17.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.060+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fb1f0003b40 con 0x7fb1fc0af900 2024-03-29T11:52:17.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.062+0000 7fb1fa7fc640 10 monclient: get_auth_request con 0x7fb1e806df20 auth_method 0 2024-03-29T11:52:17.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.062+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2667449288 0 0) 0x7fb1f00848e0 con 0x7fb1fc0af900 2024-03-29T11:52:17.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.062+0000 7fb1e3fff640 10 client.4622.objecter ms_dispatch 0x7fb1fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:52:17.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.062+0000 7fb1e3fff640 3 client.4622.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:52:17.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.062+0000 7fb1e3fff640 3 client.4622.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:52:17.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.062+0000 7fb1e3fff640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:52:17.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.063+0000 7fb1e3fff640 10 client.4622.objecter ms_handle_connect 0x7fb1e806df20 2024-03-29T11:52:17.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.064+0000 7fb20192b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:17.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.064+0000 7fb20192b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:17.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.064+0000 7fb20192b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:17.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.064+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc005080 con 0x7fb1fc0af900 2024-03-29T11:52:17.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.072+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1f00501c0 con 0x7fb1fc0af900 2024-03-29T11:52:17.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.072+0000 7fb1e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:17.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.072+0000 7fb1e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:17.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.413+0000 7fb20192b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:17.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.413+0000 7fb20192b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:17.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.413+0000 7fb20192b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:17.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.413+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1dc005330 con 0x7fb1fc0af900 2024-03-29T11:52:17.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.415+0000 7fb1e3fff640 1 -- 172.21.15.55:0/2850462787 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 824645091) 0x7fb1f00520f0 con 0x7fb1fc0af900 2024-03-29T11:52:17.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.415+0000 7fb1e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:17.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.415+0000 7fb1e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:17.422 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:52:17.422 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:52:14.925502+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:52:14.925502+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:17.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.420+0000 7fb20192b640 10 librados: watch_flush enter 2024-03-29T11:52:17.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.420+0000 7fb20192b640 10 librados: watch_flush exit 2024-03-29T11:52:17.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.420+0000 7fb20192b640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:52:17.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.420+0000 7fb20192b640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:52:17.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.420+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 >> v1:172.21.15.171:6816/1463197163 conn(0x7fb1e806df20 legacy=0x7fb1e80703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:17.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.420+0000 7fb20192b640 10 monclient: shutdown 2024-03-29T11:52:17.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.420+0000 7fb20192b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:17.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.420+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 >> v1:172.21.15.55:6790/0 conn(0x7fb1fc0af900 legacy=0x7fb1fc1d62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:17.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.421+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 shutdown_connections 2024-03-29T11:52:17.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.421+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 >> 172.21.15.55:0/2850462787 conn(0x7fb1fc0b35e0 msgr2=0x7fb1fc0ad2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:17.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.423+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 shutdown_connections 2024-03-29T11:52:17.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.423+0000 7fb20192b640 1 -- 172.21.15.55:0/2850462787 wait complete. 2024-03-29T11:52:17.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:17.423+0000 7fb20192b640 1 librados: shutdown 2024-03-29T11:52:17.448 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:24.254 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.393+0000 7ff50f67d640 10 monclient: get_monmap_and_config 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.393+0000 7ff50f67d640 10 monclient: build_initial_monmap 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.393+0000 7ff50f67d640 10 monclient: monmap: 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:52:24.394662+0000 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:52:24.394662+0000 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:24.401 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:24.402 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:52:24.402 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:52:24.402 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:24.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.395+0000 7ff50f67d640 1 Processor -- start 2024-03-29T11:52:24.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.395+0000 7ff50f67d640 1 -- start start 2024-03-29T11:52:24.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.395+0000 7ff50f67d640 10 monclient: init 2024-03-29T11:52:24.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.396+0000 7ff50f67d640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:24.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.396+0000 7ff50f67d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:52:24.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.396+0000 7ff50f67d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5081422b0 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:24.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.397+0000 7ff50f67d640 10 monclient(hunting): picked mon.noname-a con 0x7ff508142690 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:24.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.399+0000 7ff50f67d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5080b17d0 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:24.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.399+0000 7ff50f67d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff508142c80 con 0x7ff508142690 2024-03-29T11:52:24.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.399+0000 7ff50f67d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080aae40 con 0x7ff5080b17d0 2024-03-29T11:52:24.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.399+0000 7ff50f67d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50813fbd0 con 0x7ff5081422b0 2024-03-29T11:52:24.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.399+0000 7ff50f67d640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:24.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.399+0000 7ff50f67d640 10 monclient(hunting): authenticate will time out at 1557.678782s 2024-03-29T11:52:24.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.397+0000 7ff50d419640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7ff5081422b0 0x7ff50813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:45580/0 (socket says 172.21.15.55:45580) 2024-03-29T11:52:24.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.399+0000 7ff50d419640 1 -- 172.21.15.55:0/811745892 learned_addr learned my addr 172.21.15.55:0/811745892 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363147829 0 0) 0x7ff508142c80 con 0x7ff508142690 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f4003520 con 0x7ff508142690 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42586645 0 0) 0x7ff50813fbd0 con 0x7ff5081422b0 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.401+0000 7ff4fffff640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:52:24.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff508142c80 con 0x7ff5081422b0 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454006955 0 0) 0x7ff4f4003520 con 0x7ff508142690 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 >> v1:172.21.15.55:6790/0 conn(0x7ff5081422b0 legacy=0x7ff50813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 >> v1:172.21.15.171:6789/0 conn(0x7ff5080b17d0 legacy=0x7ff5080b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 10 monclient: _finish_hunting 0 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 1 monclient: found mon.noname-a 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508151940 con 0x7ff508142690 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 10 monclient: _finish_auth 0 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 10 monclient: _check_auth_tickets 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ff504001690 con 0x7ff508142690 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff50f67d640 5 monclient: authenticate success, global_id 4599 2024-03-29T11:52:24.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.402+0000 7ff50f67d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.403+0000 7ff4fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.403+0000 7ff4fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.403+0000 7ff4fffff640 10 monclient: dump: 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:24.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.403+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5040018b0 con 0x7ff508142690 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.403+0000 7ff50f67d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.403+0000 7ff4fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.403+0000 7ff4fffff640 1 -- 172.21.15.55:0/811745892 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ff504001bc0 con 0x7ff508142690 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.403+0000 7ff50f67d640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.403+0000 7ff50f67d640 4 set_mon_vals no callback set 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.404+0000 7ff50f67d640 10 monclient: shutdown 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.404+0000 7ff50f67d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.404+0000 7ff50f67d640 1 -- 172.21.15.55:0/811745892 >> v1:172.21.15.55:6789/0 conn(0x7ff508142690 legacy=0x7ff5080b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.404+0000 7ff4fffff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.404+0000 7ff50f67d640 1 -- 172.21.15.55:0/811745892 shutdown_connections 2024-03-29T11:52:24.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.404+0000 7ff50f67d640 1 -- 172.21.15.55:0/811745892 >> 172.21.15.55:0/811745892 conn(0x7ff508066a00 msgr2=0x7ff508066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:24.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.405+0000 7ff50f67d640 1 -- 172.21.15.55:0/811745892 shutdown_connections 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.406+0000 7ff50f67d640 1 -- 172.21.15.55:0/811745892 wait complete. 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.408+0000 7ff50f67d640 10 monclient: build_initial_monmap 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.408+0000 7ff50f67d640 10 monclient: monmap: 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.408+0000 7ff50f67d640 1 librados: starting msgr at 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.409+0000 7ff50f67d640 1 librados: starting objecter 2024-03-29T11:52:24.417 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.409+0000 7ff50f67d640 1 Processor -- start 2024-03-29T11:52:24.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.409+0000 7ff50f67d640 1 -- start start 2024-03-29T11:52:24.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.409+0000 7ff50f67d640 1 librados: setting wanted keys 2024-03-29T11:52:24.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.409+0000 7ff50f67d640 1 librados: calling monclient init 2024-03-29T11:52:24.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.409+0000 7ff50f67d640 10 monclient: init 2024-03-29T11:52:24.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.410+0000 7ff50f67d640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:24.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.410+0000 7ff50f67d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:52:24.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.411+0000 7ff50f67d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5080b17d0 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.411+0000 7ff50f67d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5081422b0 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff50f67d640 10 monclient(hunting): picked mon.noname-b con 0x7ff508142690 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff50f67d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080ac8e0 con 0x7ff5081422b0 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff50f67d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080abec0 con 0x7ff508142690 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff50f67d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080acb30 con 0x7ff5080b17d0 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff50f67d640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff50f67d640 10 monclient(hunting): authenticate will time out at 1557.691958s 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff50d419640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7ff5080b17d0 0x7ff5080ab7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:39094/0 (socket says 172.21.15.55:39094) 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff50d419640 1 -- 172.21.15.55:0/4291399645 learned_addr learned my addr 172.21.15.55:0/4291399645 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff4fdffb640 10 client.?.objecter ms_handle_connect 0x7ff5080b17d0 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.412+0000 7ff4fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033741179 0 0) 0x7ff5080acb30 con 0x7ff5080b17d0 2024-03-29T11:52:24.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:24.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:24.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:24.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:52:24.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ec004eb0 con 0x7ff5080b17d0 2024-03-29T11:52:24.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 183896775 0 0) 0x7ff4ec004eb0 con 0x7ff5080b17d0 2024-03-29T11:52:24.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5080acb30 con 0x7ff5080b17d0 2024-03-29T11:52:24.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ff4f8001840 con 0x7ff5080b17d0 2024-03-29T11:52:24.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 10 monclient(hunting): dump: 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.413+0000 7ff4fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:24.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.414+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 781327455 0 0) 0x7ff5080acb30 con 0x7ff5080b17d0 2024-03-29T11:52:24.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.415+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 >> v1:172.21.15.171:6789/0 conn(0x7ff508142690 legacy=0x7ff50812fde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:24.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 >> v1:172.21.15.55:6789/0 conn(0x7ff5081422b0 legacy=0x7ff5080b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:24.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 1 monclient: found mon.c 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5080ace10 con 0x7ff5080b17d0 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 10 monclient: _finish_auth 0 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 10 client.?.objecter ms_handle_connect 0x7ff5081422b0 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff4fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff50f67d640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff50f67d640 10 monclient: _renew_subs 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff50f67d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.417+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5080ad0f0 con 0x7ff5080b17d0 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.418+0000 7ff50f67d640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.418+0000 7ff50f67d640 10 monclient: _renew_subs 2024-03-29T11:52:24.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.418+0000 7ff50f67d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:24.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.418+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5080b1080 con 0x7ff5080b17d0 2024-03-29T11:52:24.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.418+0000 7ff50f67d640 1 librados: init done 2024-03-29T11:52:24.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.419+0000 7ff50f67d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:24.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.419+0000 7ff50f67d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:24.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.419+0000 7ff50f67d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:24.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.419+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff508075f10 con 0x7ff5080b17d0 2024-03-29T11:52:24.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.422+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4f8001aa0 con 0x7ff5080b17d0 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.422+0000 7ff4fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.422+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7ff4f8004c30 con 0x7ff5080b17d0 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.422+0000 7ff4fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.422+0000 7ff4fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.422+0000 7ff4fdffb640 10 monclient: dump: 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:24.430 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.423+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7ff4f8004e50 con 0x7ff5080b17d0 2024-03-29T11:52:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.423+0000 7ff4feffd640 4 set_mon_vals no callback set 2024-03-29T11:52:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.423+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2667449288 0 0) 0x7ff4f8084b70 con 0x7ff5080b17d0 2024-03-29T11:52:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.424+0000 7ff4fdffb640 10 client.4628.objecter ms_dispatch 0x7ff508000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:52:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.424+0000 7ff4fdffb640 3 client.4628.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:52:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.424+0000 7ff4fdffb640 3 client.4628.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:52:24.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.424+0000 7ff4fdffb640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:52:24.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.424+0000 7ff50cc18640 10 monclient: get_auth_request con 0x7ff4ec06db80 auth_method 0 2024-03-29T11:52:24.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.425+0000 7ff4fdffb640 10 client.4628.objecter ms_handle_connect 0x7ff4ec06db80 2024-03-29T11:52:24.437 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.430+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4f80505e0 con 0x7ff5080b17d0 2024-03-29T11:52:24.437 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.430+0000 7ff4fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:24.437 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.430+0000 7ff4fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:24.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.765+0000 7ff50f67d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:24.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.765+0000 7ff50f67d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:24.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.765+0000 7ff50f67d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:24.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.765+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5080b2270 con 0x7ff5080b17d0 2024-03-29T11:52:24.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.767+0000 7ff4fdffb640 1 -- 172.21.15.55:0/4291399645 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 824645091) 0x7ff4f8054a90 con 0x7ff5080b17d0 2024-03-29T11:52:24.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.767+0000 7ff4fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:24.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.767+0000 7ff4fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:24.778 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:52:24.778 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:52:14.925502+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:52:14.925502+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:24.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.771+0000 7ff50f67d640 10 librados: watch_flush enter 2024-03-29T11:52:24.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.771+0000 7ff50f67d640 10 librados: watch_flush exit 2024-03-29T11:52:24.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.771+0000 7ff50f67d640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:52:24.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.771+0000 7ff50f67d640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:52:24.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.771+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 >> v1:172.21.15.171:6816/1463197163 conn(0x7ff4ec06db80 legacy=0x7ff4ec070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:24.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.771+0000 7ff50f67d640 10 monclient: shutdown 2024-03-29T11:52:24.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.771+0000 7ff50f67d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:24.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.771+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 >> v1:172.21.15.55:6790/0 conn(0x7ff5080b17d0 legacy=0x7ff5080ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:24.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.772+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 shutdown_connections 2024-03-29T11:52:24.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.772+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 >> 172.21.15.55:0/4291399645 conn(0x7ff508066a00 msgr2=0x7ff5080b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:24.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.772+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 shutdown_connections 2024-03-29T11:52:24.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.772+0000 7ff50f67d640 1 -- 172.21.15.55:0/4291399645 wait complete. 2024-03-29T11:52:24.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:24.772+0000 7ff50f67d640 1 librados: shutdown 2024-03-29T11:52:24.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:31.602 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:31.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.748+0000 7fab4ef41640 10 monclient: get_monmap_and_config 2024-03-29T11:52:31.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.748+0000 7fab4ef41640 10 monclient: build_initial_monmap 2024-03-29T11:52:31.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.748+0000 7fab4ef41640 10 monclient: monmap: 2024-03-29T11:52:31.757 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:31.757 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:52:31.749246+0000 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:52:31.749246+0000 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.750+0000 7fab4ef41640 1 Processor -- start 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.750+0000 7fab4ef41640 1 -- start start 2024-03-29T11:52:31.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.750+0000 7fab4ef41640 10 monclient: init 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 10 monclient(hunting): picked mon.noname-b con 0x7fab40034850 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 10 monclient(hunting): picked mon.noname-c con 0x7fab400c51c0 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 10 monclient(hunting): picked mon.noname-a con 0x7fab400c7cd0 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab40029190 con 0x7fab400c7cd0 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab400293e0 con 0x7fab400c51c0 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab400c8340 con 0x7fab40034850 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:31.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.751+0000 7fab4ef41640 10 monclient(hunting): authenticate will time out at 1565.030992s 2024-03-29T11:52:31.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.755+0000 7fab4d73e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fab400c51c0 0x7fab40029800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:36463/0 (socket says 172.21.15.55:36463) 2024-03-29T11:52:31.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.755+0000 7fab4d73e640 1 -- 172.21.15.55:0/2898894251 learned_addr learned my addr 172.21.15.55:0/2898894251 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.755+0000 7fab4cf3d640 1 -- 172.21.15.55:0/2898894251 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780270043 0 0) 0x7fab400293e0 con 0x7fab400c51c0 2024-03-29T11:52:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.756+0000 7fab4cf3d640 10 monclient(hunting): _negotiate 2024-03-29T11:52:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.756+0000 7fab4cf3d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.756+0000 7fab4cf3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.756+0000 7fab4cf3d640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:52:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.756+0000 7fab4cf3d640 1 -- 172.21.15.55:0/2898894251 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34003520 con 0x7fab400c51c0 2024-03-29T11:52:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.756+0000 7fab4cf3d640 1 -- 172.21.15.55:0/2898894251 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2151896791 0 0) 0x7fab34003520 con 0x7fab400c51c0 2024-03-29T11:52:31.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.756+0000 7fab4cf3d640 1 -- 172.21.15.55:0/2898894251 >> v1:172.21.15.55:6790/0 conn(0x7fab40034850 legacy=0x7fab400c4ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 1 -- 172.21.15.55:0/2898894251 >> v1:172.21.15.55:6789/0 conn(0x7fab400c7cd0 legacy=0x7fab40029f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 10 monclient: _finish_hunting 0 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 1 monclient: found mon.noname-c 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 1 -- 172.21.15.55:0/2898894251 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab40032980 con 0x7fab400c51c0 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 10 monclient: _finish_auth 0 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 10 monclient: _check_auth_tickets 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4ef41640 5 monclient: authenticate success, global_id 4573 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4ef41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:31.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 1 -- 172.21.15.55:0/2898894251 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fab4804f730 con 0x7fab400c51c0 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4cf3d640 10 monclient: dump: 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:31.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.758+0000 7fab4ef41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:31.770 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.761+0000 7fab4cf3d640 1 -- 172.21.15.55:0/2898894251 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab480008d0 con 0x7fab400c51c0 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.761+0000 7fab4cf3d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.761+0000 7fab4ef41640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.761+0000 7fab4ef41640 4 set_mon_vals no callback set 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.762+0000 7fab4ef41640 10 monclient: shutdown 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.762+0000 7fab4ef41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.762+0000 7fab4ef41640 1 -- 172.21.15.55:0/2898894251 >> v1:172.21.15.171:6789/0 conn(0x7fab400c51c0 legacy=0x7fab40029800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.762+0000 7fab4cf3d640 1 -- 172.21.15.55:0/2898894251 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fab480758b0 con 0x7fab400c51c0 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.762+0000 7fab4cf3d640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.762+0000 7fab4ef41640 1 -- 172.21.15.55:0/2898894251 shutdown_connections 2024-03-29T11:52:31.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.762+0000 7fab4ef41640 1 -- 172.21.15.55:0/2898894251 >> 172.21.15.55:0/2898894251 conn(0x7fab400058f0 msgr2=0x7fab40005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:31.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.764+0000 7fab4ef41640 1 -- 172.21.15.55:0/2898894251 shutdown_connections 2024-03-29T11:52:31.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.765+0000 7fab4ef41640 1 -- 172.21.15.55:0/2898894251 wait complete. 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.766+0000 7fab4ef41640 10 monclient: build_initial_monmap 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.766+0000 7fab4ef41640 10 monclient: monmap: 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.766+0000 7fab4ef41640 1 librados: starting msgr at 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.766+0000 7fab4ef41640 1 librados: starting objecter 2024-03-29T11:52:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.766+0000 7fab4ef41640 1 Processor -- start 2024-03-29T11:52:31.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.766+0000 7fab4ef41640 1 -- start start 2024-03-29T11:52:31.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.766+0000 7fab4ef41640 1 librados: setting wanted keys 2024-03-29T11:52:31.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.766+0000 7fab4ef41640 1 librados: calling monclient init 2024-03-29T11:52:31.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.766+0000 7fab4ef41640 10 monclient: init 2024-03-29T11:52:31.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:31.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:52:31.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 10 monclient(hunting): picked mon.noname-a con 0x7fab40034850 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:31.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 10 monclient(hunting): picked mon.noname-c con 0x7fab400c51c0 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:31.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 10 monclient(hunting): picked mon.noname-b con 0x7fab400c7cd0 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:31.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab400ca7e0 con 0x7fab40034850 2024-03-29T11:52:31.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab400caac0 con 0x7fab400c7cd0 2024-03-29T11:52:31.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab400cada0 con 0x7fab400c51c0 2024-03-29T11:52:31.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:31.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.767+0000 7fab4ef41640 10 monclient(hunting): authenticate will time out at 1565.047223s 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.773+0000 7fab4d73e640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fab400c51c0 0x7fab40164670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:40191/0 (socket says 172.21.15.55:40191) 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.773+0000 7fab4d73e640 1 -- 172.21.15.55:0/2725948227 learned_addr learned my addr 172.21.15.55:0/2725948227 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.773+0000 7fab3e7fc640 10 client.?.objecter ms_handle_connect 0x7fab400c51c0 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026914401 0 0) 0x7fab400cada0 con 0x7fab400c51c0 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab2c004eb0 con 0x7fab400c51c0 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 client.?.objecter ms_handle_connect 0x7fab400c7cd0 2024-03-29T11:52:31.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:31.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983970772 0 0) 0x7fab400caac0 con 0x7fab400c7cd0 2024-03-29T11:52:31.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:52:31.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:31.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:31.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:52:31.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.774+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab400cada0 con 0x7fab400c7cd0 2024-03-29T11:52:31.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.775+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2695018039 0 0) 0x7fab2c004eb0 con 0x7fab400c51c0 2024-03-29T11:52:31.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.775+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab400caac0 con 0x7fab400c51c0 2024-03-29T11:52:31.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.775+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fab4804f400 con 0x7fab400c51c0 2024-03-29T11:52:31.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.775+0000 7fab3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:31.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.778+0000 7fab3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:31.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.778+0000 7fab3e7fc640 10 monclient(hunting): dump: 2024-03-29T11:52:31.787 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:31.787 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.778+0000 7fab3e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.778+0000 7fab3e7fc640 10 client.?.objecter ms_handle_connect 0x7fab40034850 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.778+0000 7fab3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.778+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 226297072 0 0) 0x7fab400caac0 con 0x7fab400c51c0 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.778+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 >> v1:172.21.15.171:6789/0 conn(0x7fab400c7cd0 legacy=0x7fab4016c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.778+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 >> v1:172.21.15.55:6789/0 conn(0x7fab40034850 legacy=0x7fab400cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.778+0000 7fab3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab3e7fc640 1 monclient: found mon.c 2024-03-29T11:52:31.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab3e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab400c9390 con 0x7fab400c51c0 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab3e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab4ef41640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab4ef41640 10 monclient: _renew_subs 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab4ef41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab400cb9b0 con 0x7fab400c51c0 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab4ef41640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab4ef41640 10 monclient: _renew_subs 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab4ef41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab400c9a90 con 0x7fab400c51c0 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.779+0000 7fab4ef41640 1 librados: init done 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.780+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab4804fdd0 con 0x7fab400c51c0 2024-03-29T11:52:31.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.780+0000 7fab3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:31.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.780+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fab48067bf0 con 0x7fab400c51c0 2024-03-29T11:52:31.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.780+0000 7fab3ffff640 4 set_mon_vals no callback set 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.781+0000 7fab4ef41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.781+0000 7fab4ef41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.781+0000 7fab4ef41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.781+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab30005080 con 0x7fab400c51c0 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.781+0000 7fab3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.781+0000 7fab3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.781+0000 7fab3e7fc640 10 monclient: dump: 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:31.791 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:31.792 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:31.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.781+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fab48067d50 con 0x7fab400c51c0 2024-03-29T11:52:31.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.782+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2667449288 0 0) 0x7fab480ff5e0 con 0x7fab400c51c0 2024-03-29T11:52:31.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.782+0000 7fab3e7fc640 10 client.4631.objecter ms_dispatch 0x7fab48000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:52:31.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.782+0000 7fab3e7fc640 3 client.4631.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:52:31.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.782+0000 7fab3e7fc640 3 client.4631.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:52:31.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.782+0000 7fab3e7fc640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:52:31.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.782+0000 7fab4df3f640 10 monclient: get_auth_request con 0x7fab2c06dc30 auth_method 0 2024-03-29T11:52:31.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.783+0000 7fab3e7fc640 10 client.4631.objecter ms_handle_connect 0x7fab2c06dc30 2024-03-29T11:52:31.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.790+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab480caee0 con 0x7fab400c51c0 2024-03-29T11:52:31.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.790+0000 7fab3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:31.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:31.790+0000 7fab3e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:32.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.127+0000 7fab4ef41640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:32.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.127+0000 7fab4ef41640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:32.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.127+0000 7fab4ef41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:32.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.127+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab30005330 con 0x7fab400c51c0 2024-03-29T11:52:32.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.128+0000 7fab3e7fc640 1 -- 172.21.15.55:0/2725948227 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 824645091) 0x7fab480cce10 con 0x7fab400c51c0 2024-03-29T11:52:32.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.128+0000 7fab3e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:32.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.128+0000 7fab3e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:32.136 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:52:32.136 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:52:14.925502+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:52:14.925502+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:32.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.131+0000 7fab4ef41640 10 librados: watch_flush enter 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.131+0000 7fab4ef41640 10 librados: watch_flush exit 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.131+0000 7fab4ef41640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.131+0000 7fab4ef41640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.132+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 >> v1:172.21.15.171:6816/1463197163 conn(0x7fab2c06dc30 legacy=0x7fab2c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.132+0000 7fab4ef41640 10 monclient: shutdown 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.132+0000 7fab4ef41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.132+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 >> v1:172.21.15.55:6790/0 conn(0x7fab400c51c0 legacy=0x7fab40164670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.133+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 shutdown_connections 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.133+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 >> 172.21.15.55:0/2725948227 conn(0x7fab400058f0 msgr2=0x7fab400c4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:32.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.133+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 shutdown_connections 2024-03-29T11:52:32.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.133+0000 7fab4ef41640 1 -- 172.21.15.55:0/2725948227 wait complete. 2024-03-29T11:52:32.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:32.133+0000 7fab4ef41640 1 librados: shutdown 2024-03-29T11:52:32.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:38.965 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:39.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.112+0000 7f76dbfff640 10 monclient: get_monmap_and_config 2024-03-29T11:52:39.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.112+0000 7f76dbfff640 10 monclient: build_initial_monmap 2024-03-29T11:52:39.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.112+0000 7f76dbfff640 10 monclient: monmap: 2024-03-29T11:52:39.121 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:39.121 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:39.121 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:52:39.113552+0000 2024-03-29T11:52:39.122 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:52:39.113552+0000 2024-03-29T11:52:39.122 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:39.122 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:39.122 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:39.122 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:52:39.122 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:52:39.122 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:39.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.113+0000 7f76dbfff640 1 Processor -- start 2024-03-29T11:52:39.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.113+0000 7f76dbfff640 1 -- start start 2024-03-29T11:52:39.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.113+0000 7f76dbfff640 10 monclient: init 2024-03-29T11:52:39.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.114+0000 7f76dbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:39.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.114+0000 7f76dbfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.118+0000 7f76daffd640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f76d4034850 0x7f76d40c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:49159/0 (socket says 172.21.15.55:49159) 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.118+0000 7f76daffd640 1 -- 172.21.15.55:0/3704777808 learned_addr learned my addr 172.21.15.55:0/3704777808 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.118+0000 7f76dbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f76d4034850 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.118+0000 7f76dbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f76d402a640 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.118+0000 7f76dbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f76d40327d0 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.118+0000 7f76dbfff640 1 -- 172.21.15.55:0/3704777808 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d402c820 con 0x7f76d40327d0 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.118+0000 7f76dbfff640 1 -- 172.21.15.55:0/3704777808 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d402a120 con 0x7f76d402a640 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.118+0000 7f76dbfff640 1 -- 172.21.15.55:0/3704777808 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d40c75b0 con 0x7f76d4034850 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.119+0000 7f76dbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:39.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.119+0000 7f76dbfff640 10 monclient(hunting): authenticate will time out at 1572.398686s 2024-03-29T11:52:39.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.119+0000 7f76d9ffb640 1 -- 172.21.15.55:0/3704777808 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067992772 0 0) 0x7f76d402a120 con 0x7f76d402a640 2024-03-29T11:52:39.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.119+0000 7f76d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:39.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.119+0000 7f76d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:39.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.119+0000 7f76d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:39.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.119+0000 7f76d9ffb640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:52:39.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.119+0000 7f76d9ffb640 1 -- 172.21.15.55:0/3704777808 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c4003520 con 0x7f76d402a640 2024-03-29T11:52:39.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.120+0000 7f76d9ffb640 1 -- 172.21.15.55:0/3704777808 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 303764369 0 0) 0x7f76c4003520 con 0x7f76d402a640 2024-03-29T11:52:39.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.120+0000 7f76d9ffb640 1 -- 172.21.15.55:0/3704777808 >> v1:172.21.15.55:6790/0 conn(0x7f76d4034850 legacy=0x7f76d40c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:39.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.121+0000 7f76d9ffb640 1 -- 172.21.15.55:0/3704777808 >> v1:172.21.15.55:6789/0 conn(0x7f76d40327d0 legacy=0x7f76d40cf240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:52:39.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:52:39.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:52:39.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:39.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 1 -- 172.21.15.55:0/3704777808 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4033110 con 0x7f76d402a640 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 1 -- 172.21.15.55:0/3704777808 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f76cc001690 con 0x7f76d402a640 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.123+0000 7f76d9ffb640 10 monclient: dump: 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:39.133 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.124+0000 7f76d9ffb640 1 -- 172.21.15.55:0/3704777808 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76cc001920 con 0x7f76d402a640 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.124+0000 7f76d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.124+0000 7f76d9ffb640 1 -- 172.21.15.55:0/3704777808 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f76cc001cb0 con 0x7f76d402a640 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.124+0000 7f76d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.124+0000 7f76d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.124+0000 7f76d9ffb640 10 monclient: dump: 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:39.134 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.124+0000 7f76dbfff640 5 monclient: authenticate success, global_id 4579 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.124+0000 7f76dbfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.124+0000 7f76dbfff640 4 set_mon_vals no callback set 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.125+0000 7f76dbfff640 10 monclient: shutdown 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.125+0000 7f76dbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.125+0000 7f76dbfff640 1 -- 172.21.15.55:0/3704777808 >> v1:172.21.15.171:6789/0 conn(0x7f76d402a640 legacy=0x7f76d40320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.125+0000 7f76dbfff640 1 -- 172.21.15.55:0/3704777808 shutdown_connections 2024-03-29T11:52:39.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.125+0000 7f76dbfff640 1 -- 172.21.15.55:0/3704777808 >> 172.21.15.55:0/3704777808 conn(0x7f76d40058f0 msgr2=0x7f76d4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:39.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.127+0000 7f76dbfff640 1 -- 172.21.15.55:0/3704777808 shutdown_connections 2024-03-29T11:52:39.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.129+0000 7f76dbfff640 1 -- 172.21.15.55:0/3704777808 wait complete. 2024-03-29T11:52:39.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.130+0000 7f76dbfff640 10 monclient: build_initial_monmap 2024-03-29T11:52:39.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.130+0000 7f76dbfff640 10 monclient: monmap: 2024-03-29T11:52:39.138 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:39.138 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:39.138 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:52:39.138 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:52:39.138 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:39.138 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:39.138 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:39.139 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:52:39.139 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:52:39.139 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:39.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.131+0000 7f76dbfff640 1 librados: starting msgr at 2024-03-29T11:52:39.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.131+0000 7f76dbfff640 1 librados: starting objecter 2024-03-29T11:52:39.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.131+0000 7f76dbfff640 1 Processor -- start 2024-03-29T11:52:39.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.131+0000 7f76dbfff640 1 -- start start 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.131+0000 7f76dbfff640 1 librados: setting wanted keys 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.131+0000 7f76dbfff640 1 librados: calling monclient init 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.131+0000 7f76dbfff640 10 monclient: init 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.132+0000 7f76dbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.132+0000 7f76dbfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.132+0000 7f76dbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f76d402a640 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.135+0000 7f76dbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f76d40327d0 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.135+0000 7f76dbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f76d4034850 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.135+0000 7f76dbfff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d4031d30 con 0x7f76d40327d0 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.135+0000 7f76dbfff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d4031ec0 con 0x7f76d4034850 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.135+0000 7f76dbfff640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d4031290 con 0x7f76d402a640 2024-03-29T11:52:39.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.135+0000 7f76dbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:39.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.135+0000 7f76dbfff640 10 monclient(hunting): authenticate will time out at 1572.415447s 2024-03-29T11:52:39.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76db7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f76d4034850 0x7f76d4161650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:41006/0 (socket says 172.21.15.55:41006) 2024-03-29T11:52:39.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76db7fe640 1 -- 172.21.15.55:0/2545119866 learned_addr learned my addr 172.21.15.55:0/2545119866 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:39.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 10 client.?.objecter ms_handle_connect 0x7f76d4034850 2024-03-29T11:52:39.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:39.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117541137 0 0) 0x7f76d4031ec0 con 0x7f76d4034850 2024-03-29T11:52:39.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:39.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:39.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:39.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:52:39.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c8004eb0 con 0x7f76d4034850 2024-03-29T11:52:39.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 10 client.?.objecter ms_handle_connect 0x7f76d40327d0 2024-03-29T11:52:39.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.139+0000 7f76c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:39.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.140+0000 7f76c3fff640 10 client.?.objecter ms_handle_connect 0x7f76d402a640 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.140+0000 7f76c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.140+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2712649906 0 0) 0x7f76c8004eb0 con 0x7f76d4034850 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.140+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76d4031ec0 con 0x7f76d4034850 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.140+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f76d00018b0 con 0x7f76d4034850 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.140+0000 7f76c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.140+0000 7f76c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.140+0000 7f76c3fff640 10 monclient(hunting): dump: 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:39.148 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:39.149 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:39.149 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:39.149 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:39.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.140+0000 7f76c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:39.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 687025130 0 0) 0x7f76d4031ec0 con 0x7f76d4034850 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 >> v1:172.21.15.55:6790/0 conn(0x7f76d402a640 legacy=0x7f76d40304c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 >> v1:172.21.15.55:6789/0 conn(0x7f76d40327d0 legacy=0x7f76d4160f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 1 monclient: found mon.b 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4031420 con 0x7f76d4034850 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76dbfff640 5 monclient: authenticate success, global_id 4582 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76dbfff640 10 monclient: _renew_subs 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d416c990 con 0x7f76d4034850 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76dbfff640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:39.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76dbfff640 10 monclient: _renew_subs 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d416cd00 con 0x7f76d4034850 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.141+0000 7f76dbfff640 1 librados: init done 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.142+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76d00035f0 con 0x7f76d4034850 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.142+0000 7f76c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.142+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f76d0001a10 con 0x7f76d4034850 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.142+0000 7f76c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.142+0000 7f76c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.142+0000 7f76c3fff640 10 monclient: dump: 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:39.151 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:39.152 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:39.152 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:39.152 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:39.152 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:39.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.143+0000 7f76d8ff9640 4 set_mon_vals no callback set 2024-03-29T11:52:39.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.144+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f76d0003ac0 con 0x7f76d4034850 2024-03-29T11:52:39.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.145+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2667449288 0 0) 0x7f76d0084a40 con 0x7f76d4034850 2024-03-29T11:52:39.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.145+0000 7f76c3fff640 10 client.4582.objecter ms_dispatch 0x7f76dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:52:39.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.145+0000 7f76c3fff640 3 client.4582.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:52:39.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.145+0000 7f76c3fff640 3 client.4582.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:52:39.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.145+0000 7f76c3fff640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-29T11:52:39.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.146+0000 7f76daffd640 10 monclient: get_auth_request con 0x7f76c806dc80 auth_method 0 2024-03-29T11:52:39.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.146+0000 7f76c3fff640 10 client.4582.objecter ms_handle_connect 0x7f76c806dc80 2024-03-29T11:52:39.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.148+0000 7f76dbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:39.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.148+0000 7f76dbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:39.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.148+0000 7f76dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:39.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.148+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc005080 con 0x7f76d4034850 2024-03-29T11:52:39.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.155+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76d0050320 con 0x7f76d4034850 2024-03-29T11:52:39.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.155+0000 7f76c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:39.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.155+0000 7f76c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:39.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.501+0000 7f76dbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:39.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.501+0000 7f76dbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.501+0000 7f76dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:52:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.501+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76bc005330 con 0x7f76d4034850 2024-03-29T11:52:39.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.503+0000 7f76c3fff640 1 -- 172.21.15.55:0/2545119866 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 824645091) 0x7f76d0052250 con 0x7f76d4034850 2024-03-29T11:52:39.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.503+0000 7f76c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:39.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.503+0000 7f76c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:39.511 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:52:39.512 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:52:14.925502+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:52:14.925502+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.507+0000 7f76dbfff640 10 librados: watch_flush enter 2024-03-29T11:52:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.507+0000 7f76dbfff640 10 librados: watch_flush exit 2024-03-29T11:52:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.507+0000 7f76dbfff640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-29T11:52:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.507+0000 7f76dbfff640 10 client.4582.objecter successfully canceled tick 2024-03-29T11:52:39.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.508+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 >> v1:172.21.15.171:6816/1463197163 conn(0x7f76c806dc80 legacy=0x7f76c8070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:39.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.508+0000 7f76dbfff640 10 monclient: shutdown 2024-03-29T11:52:39.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.508+0000 7f76dbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:39.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.508+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 >> v1:172.21.15.171:6789/0 conn(0x7f76d4034850 legacy=0x7f76d4161650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:39.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.510+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 shutdown_connections 2024-03-29T11:52:39.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.510+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 >> 172.21.15.55:0/2545119866 conn(0x7f76d40058f0 msgr2=0x7f76d40043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.510+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 shutdown_connections 2024-03-29T11:52:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.511+0000 7f76dbfff640 1 -- 172.21.15.55:0/2545119866 wait complete. 2024-03-29T11:52:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:39.511+0000 7f76dbfff640 1 librados: shutdown 2024-03-29T11:52:39.537 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:46.343 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:46.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.483+0000 7f421ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:52:46.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.483+0000 7f421ffff640 10 monclient: build_initial_monmap 2024-03-29T11:52:46.491 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.483+0000 7f421ffff640 10 monclient: monmap: 2024-03-29T11:52:46.491 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:46.491 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:46.491 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:52:46.484989+0000 2024-03-29T11:52:46.491 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:52:46.484989+0000 2024-03-29T11:52:46.492 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:46.492 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:46.492 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:46.492 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:52:46.492 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:52:46.492 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:46.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.484+0000 7f421ffff640 1 Processor -- start 2024-03-29T11:52:46.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.484+0000 7f421ffff640 1 -- start start 2024-03-29T11:52:46.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.484+0000 7f421ffff640 10 monclient: init 2024-03-29T11:52:46.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:46.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:52:46.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f42200b2800 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:46.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f42200a9c00 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:46.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f42200b38c0 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4220141eb0 con 0x7f42200b38c0 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42200b22d0 con 0x7f42200b2800 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f42200b2800 0x7f4220142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:50708/0 (socket says 172.21.15.55:50708) 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421effd640 1 -- 172.21.15.55:0/2606637379 learned_addr learned my addr 172.21.15.55:0/2606637379 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 1 -- 172.21.15.55:0/2606637379 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4220142d30 con 0x7f42200a9c00 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421ffff640 10 monclient(hunting): authenticate will time out at 1579.766135s 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.486+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077848075 0 0) 0x7f42200b22d0 con 0x7f42200b2800 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208003520 con 0x7f42200b2800 2024-03-29T11:52:46.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139699490 0 0) 0x7f4220141eb0 con 0x7f42200b38c0 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient(hunting): my global_id is 4614 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208003840 con 0x7f42200b38c0 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251431283 0 0) 0x7f4208003840 con 0x7f42200b38c0 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 >> v1:172.21.15.55:6790/0 conn(0x7f42200a9c00 legacy=0x7f422013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 >> v1:172.21.15.171:6789/0 conn(0x7f42200b2800 legacy=0x7f4220142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 1 monclient: found mon.noname-a 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42200aaf90 con 0x7f42200b38c0 2024-03-29T11:52:46.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient: _finish_auth 0 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4218001970 con 0x7f42200b38c0 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421dffb640 10 monclient: dump: 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:46.499 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421ffff640 5 monclient: authenticate success, global_id 4614 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.487+0000 7f421ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.488+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4218001b50 con 0x7f42200b38c0 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.488+0000 7f421dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.488+0000 7f421ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.488+0000 7f421ffff640 4 set_mon_vals no callback set 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.488+0000 7f421dffb640 1 -- 172.21.15.55:0/2606637379 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4218001970 con 0x7f42200b38c0 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.488+0000 7f421dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.488+0000 7f421dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.488+0000 7f421dffb640 10 monclient: dump: 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:46.500 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.489+0000 7f421ffff640 10 monclient: shutdown 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.489+0000 7f421ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.489+0000 7f421ffff640 1 -- 172.21.15.55:0/2606637379 >> v1:172.21.15.55:6789/0 conn(0x7f42200b38c0 legacy=0x7f422014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.489+0000 7f421ffff640 1 -- 172.21.15.55:0/2606637379 shutdown_connections 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.489+0000 7f421ffff640 1 -- 172.21.15.55:0/2606637379 >> 172.21.15.55:0/2606637379 conn(0x7f4220066a00 msgr2=0x7f4220066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.489+0000 7f421ffff640 1 -- 172.21.15.55:0/2606637379 shutdown_connections 2024-03-29T11:52:46.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.489+0000 7f421ffff640 1 -- 172.21.15.55:0/2606637379 wait complete. 2024-03-29T11:52:46.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.490+0000 7f421ffff640 10 monclient: build_initial_monmap 2024-03-29T11:52:46.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.490+0000 7f421ffff640 10 monclient: monmap: 2024-03-29T11:52:46.509 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:46.509 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:46.509 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:52:46.509 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:52:46.509 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:46.509 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:46.509 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.490+0000 7f421ffff640 1 librados: starting msgr at 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.490+0000 7f421ffff640 1 librados: starting objecter 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.491+0000 7f421ffff640 1 Processor -- start 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.491+0000 7f421ffff640 1 -- start start 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.491+0000 7f421ffff640 1 librados: setting wanted keys 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.491+0000 7f421ffff640 1 librados: calling monclient init 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.491+0000 7f421ffff640 10 monclient: init 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f42200a9c00 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f42200b2800 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:46.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f42200b38c0 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42201cc950 con 0x7f42200b2800 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42201ccba0 con 0x7f42200a9c00 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42201cce80 con 0x7f42200b38c0 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421ffff640 10 monclient(hunting): authenticate will time out at 1579.772208s 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421e7fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f42200b2800 0x7f422013ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:34394/0 (socket says 172.21.15.55:34394) 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f421e7fc640 1 -- 172.21.15.55:0/726055581 learned_addr learned my addr 172.21.15.55:0/726055581 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f42077fe640 10 client.?.objecter ms_handle_connect 0x7f42200b2800 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.492+0000 7f42077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384117719 0 0) 0x7f42201cc950 con 0x7f42200b2800 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:46.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 10 monclient(hunting): my global_id is 4617 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f420c004eb0 con 0x7f42200b2800 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 10 client.?.objecter ms_handle_connect 0x7f42200a9c00 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 10 client.?.objecter ms_handle_connect 0x7f42200b38c0 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.493+0000 7f42077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1418966665 0 0) 0x7f420c004eb0 con 0x7f42200b2800 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42201cc950 con 0x7f42200b2800 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4214001970 con 0x7f42200b2800 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient(hunting): dump: 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:46.512 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966503866 0 0) 0x7f42201cce80 con 0x7f42200b38c0 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f420c004eb0 con 0x7f42200b38c0 2024-03-29T11:52:46.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4272579095 0 0) 0x7f42201cc950 con 0x7f42200b2800 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 >> v1:172.21.15.55:6790/0 conn(0x7f42200b38c0 legacy=0x7f422013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 >> v1:172.21.15.171:6789/0 conn(0x7f42200a9c00 legacy=0x7f4220137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 monclient: found mon.a 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42201cd160 con 0x7f42200b2800 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient: _finish_auth 0 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f42077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f421ffff640 5 monclient: authenticate success, global_id 4617 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f421ffff640 10 monclient: _renew_subs 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f421ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:52:46.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422013c7b0 con 0x7f42200b2800 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f421ffff640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f421ffff640 10 monclient: _renew_subs 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f421ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4220138b70 con 0x7f42200b2800 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.494+0000 7f421ffff640 1 librados: init done 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.495+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4214001d30 con 0x7f42200b2800 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.495+0000 7f42077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.495+0000 7f421d7fa640 4 set_mon_vals no callback set 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.495+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f4214003a60 con 0x7f42200b2800 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.495+0000 7f42077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.495+0000 7f42077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.495+0000 7f42077fe640 10 monclient: dump: 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:46.515 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.496+0000 7f421ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.496+0000 7f421ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.496+0000 7f421ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.496+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4200005080 con 0x7f42200b2800 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.497+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f4214003c40 con 0x7f42200b2800 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.498+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2667449288 0 0) 0x7f4214084b50 con 0x7f42200b2800 2024-03-29T11:52:46.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.498+0000 7f42077fe640 10 client.4617.objecter ms_dispatch 0x7f4220000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:52:46.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.498+0000 7f42077fe640 3 client.4617.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:52:46.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.498+0000 7f42077fe640 3 client.4617.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:52:46.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.498+0000 7f42077fe640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-29T11:52:46.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.498+0000 7f421effd640 10 monclient: get_auth_request con 0x7f420c06ddd0 auth_method 0 2024-03-29T11:52:46.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.500+0000 7f42077fe640 10 client.4617.objecter ms_handle_connect 0x7f420c06ddd0 2024-03-29T11:52:46.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.504+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4214050430 con 0x7f42200b2800 2024-03-29T11:52:46.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.504+0000 7f42077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:46.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.504+0000 7f42077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:46.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.907+0000 7f421ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:46.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.907+0000 7f421ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:46.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.907+0000 7f421ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-29T11:52:46.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.907+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4200005330 con 0x7f42200b2800 2024-03-29T11:52:46.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.908+0000 7f42077fe640 1 -- 172.21.15.55:0/726055581 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 824645091) 0x7f4214052360 con 0x7f42200b2800 2024-03-29T11:52:46.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.908+0000 7f42077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:46.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.908+0000 7f42077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:46.916 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:52:46.916 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:52:14.925502+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:52:14.925502+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.911+0000 7f421ffff640 10 librados: watch_flush enter 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.911+0000 7f421ffff640 10 librados: watch_flush exit 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.911+0000 7f421ffff640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.911+0000 7f421ffff640 10 client.4617.objecter successfully canceled tick 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.912+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 >> v1:172.21.15.171:6816/1463197163 conn(0x7f420c06ddd0 legacy=0x7f420c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.912+0000 7f421ffff640 10 monclient: shutdown 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.912+0000 7f421ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.912+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 >> v1:172.21.15.55:6789/0 conn(0x7f42200b2800 legacy=0x7f422013ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.912+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 shutdown_connections 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.912+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 >> 172.21.15.55:0/726055581 conn(0x7f4220066a00 msgr2=0x7f42200b7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:46.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.912+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 shutdown_connections 2024-03-29T11:52:46.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.912+0000 7f421ffff640 1 -- 172.21.15.55:0/726055581 wait complete. 2024-03-29T11:52:46.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:46.912+0000 7f421ffff640 1 librados: shutdown 2024-03-29T11:52:46.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:53.742 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:53.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.887+0000 7f8e26cba640 10 monclient: get_monmap_and_config 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.887+0000 7f8e26cba640 10 monclient: build_initial_monmap 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.887+0000 7f8e26cba640 10 monclient: monmap: 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:52:53.888354+0000 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:52:53.888354+0000 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:52:53.895 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:53.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.890+0000 7f8e26cba640 1 Processor -- start 2024-03-29T11:52:53.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.891+0000 7f8e26cba640 1 -- start start 2024-03-29T11:52:53.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.891+0000 7f8e26cba640 10 monclient: init 2024-03-29T11:52:53.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.892+0000 7f8e26cba640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:53.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.892+0000 7f8e26cba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:52:53.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.892+0000 7f8e26cba640 10 monclient(hunting): picked mon.noname-b con 0x7f8e18006a70 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:53.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.892+0000 7f8e26cba640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1802b100 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:53.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.892+0000 7f8e26cba640 10 monclient(hunting): picked mon.noname-a con 0x7f8e18029c30 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:53.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.893+0000 7f8e26cba640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1802a270 con 0x7f8e18029c30 2024-03-29T11:52:53.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.893+0000 7f8e26cba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1802c440 con 0x7f8e1802b100 2024-03-29T11:52:53.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.893+0000 7f8e26cba640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e180c9440 con 0x7f8e18006a70 2024-03-29T11:52:53.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.893+0000 7f8e26cba640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:53.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.893+0000 7f8e26cba640 10 monclient(hunting): authenticate will time out at 1587.172716s 2024-03-29T11:52:53.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.897+0000 7f8e264b9640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f8e18029c30 0x7f8e180d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54305/0 (socket says 172.21.15.55:54305) 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.897+0000 7f8e264b9640 1 -- 172.21.15.55:0/4111404462 learned_addr learned my addr 172.21.15.55:0/4111404462 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844825830 0 0) 0x7f8e180c9440 con 0x7f8e18006a70 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient(hunting): _negotiate 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c003520 con 0x7f8e18006a70 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157606376 0 0) 0x7f8e1802a270 con 0x7f8e18029c30 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient(hunting): _negotiate 2024-03-29T11:52:53.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient(hunting): my global_id is 4620 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e180c9440 con 0x7f8e18029c30 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4219197362 0 0) 0x7f8e180c9440 con 0x7f8e18029c30 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 >> v1:172.21.15.55:6790/0 conn(0x7f8e18006a70 legacy=0x7f8e180c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 >> v1:172.21.15.171:6789/0 conn(0x7f8e1802b100 legacy=0x7f8e180cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient: _finish_hunting 0 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 monclient: found mon.noname-a 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e180c95d0 con 0x7f8e18029c30 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient: _finish_auth 0 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient: _check_auth_tickets 2024-03-29T11:52:53.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8e2004f730 con 0x7f8e18029c30 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e24cb6640 10 monclient: dump: 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:53.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e26cba640 5 monclient: authenticate success, global_id 4620 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.899+0000 7f8e26cba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.900+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e20075f10 con 0x7f8e18029c30 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.900+0000 7f8e24cb6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.900+0000 7f8e24cb6640 1 -- 172.21.15.55:0/4111404462 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8e200758b0 con 0x7f8e18029c30 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.900+0000 7f8e24cb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.900+0000 7f8e24cb6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.900+0000 7f8e24cb6640 10 monclient: dump: 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:53.911 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:53.912 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:53.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.900+0000 7f8e26cba640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:53.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.900+0000 7f8e26cba640 4 set_mon_vals no callback set 2024-03-29T11:52:53.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.901+0000 7f8e26cba640 10 monclient: shutdown 2024-03-29T11:52:53.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.901+0000 7f8e26cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:53.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.901+0000 7f8e26cba640 1 -- 172.21.15.55:0/4111404462 >> v1:172.21.15.55:6789/0 conn(0x7f8e18029c30 legacy=0x7f8e180d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:53.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.904+0000 7f8e26cba640 1 -- 172.21.15.55:0/4111404462 shutdown_connections 2024-03-29T11:52:53.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.904+0000 7f8e26cba640 1 -- 172.21.15.55:0/4111404462 >> 172.21.15.55:0/4111404462 conn(0x7f8e180058f0 msgr2=0x7f8e18005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:53.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.908+0000 7f8e26cba640 1 -- 172.21.15.55:0/4111404462 shutdown_connections 2024-03-29T11:52:53.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.908+0000 7f8e26cba640 1 -- 172.21.15.55:0/4111404462 wait complete. 2024-03-29T11:52:53.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.909+0000 7f8e26cba640 10 monclient: build_initial_monmap 2024-03-29T11:52:53.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.909+0000 7f8e26cba640 10 monclient: monmap: 2024-03-29T11:52:53.917 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:52:53.917 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:53.917 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:52:53.917 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.909+0000 7f8e26cba640 1 librados: starting msgr at 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.909+0000 7f8e26cba640 1 librados: starting objecter 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.909+0000 7f8e26cba640 1 Processor -- start 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.909+0000 7f8e26cba640 1 -- start start 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.909+0000 7f8e26cba640 1 librados: setting wanted keys 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.909+0000 7f8e26cba640 1 librados: calling monclient init 2024-03-29T11:52:53.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.909+0000 7f8e26cba640 10 monclient: init 2024-03-29T11:52:53.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.910+0000 7f8e26cba640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:53.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.910+0000 7f8e26cba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:52:53.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.910+0000 7f8e26cba640 10 monclient(hunting): picked mon.noname-b con 0x7f8e18006a70 addr v1:172.21.15.171:6789/0 2024-03-29T11:52:53.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.914+0000 7f8e26cba640 10 monclient(hunting): picked mon.noname-c con 0x7f8e18029c30 addr v1:172.21.15.55:6790/0 2024-03-29T11:52:53.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e26cba640 10 monclient(hunting): picked mon.noname-a con 0x7f8e180caea0 addr v1:172.21.15.55:6789/0 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e26cba640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e180cb4d0 con 0x7f8e180caea0 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e26cba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e180cbde0 con 0x7f8e18006a70 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e26cba640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e180cc0c0 con 0x7f8e18029c30 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e26cba640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e26cba640 10 monclient(hunting): authenticate will time out at 1587.195991s 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e254b7640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f8e18029c30 0x7f8e18031bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:52519/0 (socket says 172.21.15.55:52519) 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e254b7640 1 -- 172.21.15.55:0/201696156 learned_addr learned my addr 172.21.15.55:0/201696156 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e167fc640 10 client.?.objecter ms_handle_connect 0x7f8e18029c30 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.916+0000 7f8e25cb8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8e18006a70 0x7f8e180ca790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:53355/0 (socket says 172.21.15.55:53355) 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.917+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499449239 0 0) 0x7f8e180cc0c0 con 0x7f8e18029c30 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.917+0000 7f8e167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.917+0000 7f8e167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:53.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.917+0000 7f8e167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.917+0000 7f8e167fc640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.917+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e04004eb0 con 0x7f8e18029c30 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 client.?.objecter ms_handle_connect 0x7f8e18006a70 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2581137920 0 0) 0x7f8e04004eb0 con 0x7f8e18029c30 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e180cc0c0 con 0x7f8e18029c30 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8e1c001930 con 0x7f8e18029c30 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 monclient(hunting): dump: 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:53.928 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499723113 0 0) 0x7f8e180cbde0 con 0x7f8e18006a70 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e04004eb0 con 0x7f8e18006a70 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.918+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 226988926 0 0) 0x7f8e180cc0c0 con 0x7f8e18029c30 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.919+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 >> v1:172.21.15.171:6789/0 conn(0x7f8e18006a70 legacy=0x7f8e180ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:53.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.919+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 >> v1:172.21.15.55:6789/0 conn(0x7f8e180caea0 legacy=0x7f8e1816cac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:53.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:52:53.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e167fc640 1 monclient: found mon.c 2024-03-29T11:52:53.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:53.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:53.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e180cc3a0 con 0x7f8e18029c30 2024-03-29T11:52:53.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e167fc640 10 monclient: _finish_auth 0 2024-03-29T11:52:53.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:52:53.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:53.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e1c0035f0 con 0x7f8e18029c30 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e26cba640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.920+0000 7f8e26cba640 10 monclient: _renew_subs 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.921+0000 7f8e26cba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.921+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e180322d0 con 0x7f8e18029c30 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.921+0000 7f8e26cba640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.921+0000 7f8e26cba640 10 monclient: _renew_subs 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.921+0000 7f8e26cba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.921+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1816d1d0 con 0x7f8e18029c30 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.921+0000 7f8e26cba640 1 librados: init done 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.921+0000 7f8e167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.921+0000 7f8e177fe640 4 set_mon_vals no callback set 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.922+0000 7f8e26cba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.923+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f8e1c003980 con 0x7f8e18029c30 2024-03-29T11:52:53.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.923+0000 7f8e167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.923+0000 7f8e167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.923+0000 7f8e167fc640 10 monclient: dump: 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.923+0000 7f8e26cba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.923+0000 7f8e26cba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:53.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.923+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e08005080 con 0x7f8e18029c30 2024-03-29T11:52:53.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.924+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f8e1c003ba0 con 0x7f8e18029c30 2024-03-29T11:52:53.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.925+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2667449288 0 0) 0x7f8e1c084a00 con 0x7f8e18029c30 2024-03-29T11:52:53.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.925+0000 7f8e167fc640 10 client.4646.objecter ms_dispatch 0x7f8e20000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:52:53.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.925+0000 7f8e167fc640 3 client.4646.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:52:53.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.925+0000 7f8e167fc640 3 client.4646.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:52:53.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.925+0000 7f8e167fc640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T11:52:53.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.925+0000 7f8e25cb8640 10 monclient: get_auth_request con 0x7f8e0406dd60 auth_method 0 2024-03-29T11:52:53.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.926+0000 7f8e167fc640 10 client.4646.objecter ms_handle_connect 0x7f8e0406dd60 2024-03-29T11:52:53.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.931+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e1c0502e0 con 0x7f8e18029c30 2024-03-29T11:52:53.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.931+0000 7f8e167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:53.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:53.931+0000 7f8e167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:54.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.312+0000 7f8e26cba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:54.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.312+0000 7f8e26cba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:54.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.312+0000 7f8e26cba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:52:54.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.312+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e08005330 con 0x7f8e18029c30 2024-03-29T11:52:54.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.313+0000 7f8e167fc640 1 -- 172.21.15.55:0/201696156 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 824645091) 0x7f8e1c052210 con 0x7f8e18029c30 2024-03-29T11:52:54.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.313+0000 7f8e167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:54.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.313+0000 7f8e167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:54.321 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:52:54.321 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:52:14.925502+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:52:14.925502+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:54.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 10 librados: watch_flush enter 2024-03-29T11:52:54.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 10 librados: watch_flush exit 2024-03-29T11:52:54.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T11:52:54.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 10 client.4646.objecter successfully canceled tick 2024-03-29T11:52:54.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 >> v1:172.21.15.171:6816/1463197163 conn(0x7f8e0406dd60 legacy=0x7f8e04070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:54.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 10 monclient: shutdown 2024-03-29T11:52:54.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:54.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 >> v1:172.21.15.55:6790/0 conn(0x7f8e18029c30 legacy=0x7f8e18031bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:54.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 shutdown_connections 2024-03-29T11:52:54.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.317+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 >> 172.21.15.55:0/201696156 conn(0x7f8e180058f0 msgr2=0x7f8e180372e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:54.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.318+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 shutdown_connections 2024-03-29T11:52:54.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.318+0000 7f8e26cba640 1 -- 172.21.15.55:0/201696156 wait complete. 2024-03-29T11:52:54.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:52:54.318+0000 7f8e26cba640 1 librados: shutdown 2024-03-29T11:52:54.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:01.148 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:01.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.290+0000 7fd522990640 10 monclient: get_monmap_and_config 2024-03-29T11:53:01.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.290+0000 7fd522990640 10 monclient: build_initial_monmap 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.290+0000 7fd522990640 10 monclient: monmap: 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:53:01.291418+0000 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:53:01.291418+0000 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:53:01.298 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.291+0000 7fd522990640 1 Processor -- start 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.291+0000 7fd522990640 1 -- start start 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.291+0000 7fd522990640 10 monclient: init 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 10 monclient(hunting): picked mon.noname-a con 0x7fd51c0b1f60 addr v1:172.21.15.55:6789/0 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 10 monclient(hunting): picked mon.noname-b con 0x7fd51c071eb0 addr v1:172.21.15.55:6790/0 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 10 monclient(hunting): picked mon.noname-c con 0x7fd51c143d10 addr v1:172.21.15.171:6789/0 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c06ef80 con 0x7fd51c0b1f60 2024-03-29T11:53:01.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c06f260 con 0x7fd51c143d10 2024-03-29T11:53:01.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c0b40c0 con 0x7fd51c071eb0 2024-03-29T11:53:01.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:01.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.292+0000 7fd522990640 10 monclient(hunting): authenticate will time out at 1594.572393s 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.296+0000 7fd520f2d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd51c143d10 0x7fd51c146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:35385/0 (socket says 172.21.15.55:35385) 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.296+0000 7fd520f2d640 1 -- 172.21.15.55:0/3666474113 learned_addr learned my addr 172.21.15.55:0/3666474113 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.296+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087436361 0 0) 0x7fd51c06f260 con 0x7fd51c143d10 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.296+0000 7fd51affd640 10 monclient(hunting): _negotiate 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.296+0000 7fd51affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.296+0000 7fd51affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.296+0000 7fd51affd640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.296+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd504003520 con 0x7fd51c143d10 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.297+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473177098 0 0) 0x7fd51c0b40c0 con 0x7fd51c071eb0 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.297+0000 7fd51affd640 10 monclient(hunting): _negotiate 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.297+0000 7fd51affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.297+0000 7fd51affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.297+0000 7fd51affd640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.297+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c06f260 con 0x7fd51c071eb0 2024-03-29T11:53:01.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2301996434 0 0) 0x7fd51c06f260 con 0x7fd51c071eb0 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 >> v1:172.21.15.171:6789/0 conn(0x7fd51c143d10 legacy=0x7fd51c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 >> v1:172.21.15.55:6789/0 conn(0x7fd51c0b1f60 legacy=0x7fd51c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: _finish_hunting 0 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 1 monclient: found mon.noname-b 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c155d70 con 0x7fd51c071eb0 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: _finish_auth 0 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: _check_auth_tickets 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd522990640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd522990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd510001670 con 0x7fd51c071eb0 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:53:01.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: dump: 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd522990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5100017d0 con 0x7fd51c071eb0 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 1 -- 172.21.15.55:0/3666474113 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd510001c20 con 0x7fd51c071eb0 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:01.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd51affd640 10 monclient: dump: 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd522990640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.298+0000 7fd522990640 4 set_mon_vals no callback set 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.299+0000 7fd522990640 10 monclient: shutdown 2024-03-29T11:53:01.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.299+0000 7fd522990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:01.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.299+0000 7fd522990640 1 -- 172.21.15.55:0/3666474113 >> v1:172.21.15.55:6790/0 conn(0x7fd51c071eb0 legacy=0x7fd51c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:01.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.299+0000 7fd522990640 1 -- 172.21.15.55:0/3666474113 shutdown_connections 2024-03-29T11:53:01.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.299+0000 7fd522990640 1 -- 172.21.15.55:0/3666474113 >> 172.21.15.55:0/3666474113 conn(0x7fd51c066a00 msgr2=0x7fd51c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.302+0000 7fd522990640 1 -- 172.21.15.55:0/3666474113 shutdown_connections 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.302+0000 7fd522990640 1 -- 172.21.15.55:0/3666474113 wait complete. 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.303+0000 7fd522990640 10 monclient: build_initial_monmap 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.303+0000 7fd522990640 10 monclient: monmap: 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:01.312 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.303+0000 7fd522990640 1 librados: starting msgr at 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.303+0000 7fd522990640 1 librados: starting objecter 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.303+0000 7fd522990640 1 Processor -- start 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 1 -- start start 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 1 librados: setting wanted keys 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 1 librados: calling monclient init 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 10 monclient: init 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 10 monclient(hunting): picked mon.noname-b con 0x7fd51c0b1f60 addr v1:172.21.15.171:6789/0 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 10 monclient(hunting): picked mon.noname-a con 0x7fd51c143d10 addr v1:172.21.15.55:6789/0 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 10 monclient(hunting): picked mon.noname-c con 0x7fd51c142e60 addr v1:172.21.15.55:6790/0 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c154660 con 0x7fd51c143d10 2024-03-29T11:53:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c1423c0 con 0x7fd51c0b1f60 2024-03-29T11:53:01.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c1dfa70 con 0x7fd51c142e60 2024-03-29T11:53:01.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:01.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.304+0000 7fd522990640 10 monclient(hunting): authenticate will time out at 1594.584533s 2024-03-29T11:53:01.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd51bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd51c0b1f60 0x7fd51c1dec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.55:35481/0 (socket says 172.21.15.55:35481) 2024-03-29T11:53:01.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd51bfff640 1 -- 172.21.15.55:0/298572894 learned_addr learned my addr 172.21.15.55:0/298572894 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:53:01.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd518ff9640 10 client.?.objecter ms_handle_connect 0x7fd51c0b1f60 2024-03-29T11:53:01.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd518ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846818119 0 0) 0x7fd51c1423c0 con 0x7fd51c0b1f60 2024-03-29T11:53:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd518ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:53:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd518ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd518ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd518ff9640 10 monclient(hunting): my global_id is 4597 2024-03-29T11:53:01.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.310+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd508004eb0 con 0x7fd51c0b1f60 2024-03-29T11:53:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.311+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1264321633 0 0) 0x7fd508004eb0 con 0x7fd51c0b1f60 2024-03-29T11:53:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.311+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd51c1423c0 con 0x7fd51c0b1f60 2024-03-29T11:53:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.311+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd50c001960 con 0x7fd51c0b1f60 2024-03-29T11:53:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.311+0000 7fd518ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.311+0000 7fd518ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:53:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.311+0000 7fd518ff9640 10 monclient(hunting): dump: 2024-03-29T11:53:01.319 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.311+0000 7fd518ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.312+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2035004212 0 0) 0x7fd51c1423c0 con 0x7fd51c0b1f60 2024-03-29T11:53:01.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.312+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 >> v1:172.21.15.55:6790/0 conn(0x7fd51c142e60 legacy=0x7fd51c1df360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:53:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.313+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 >> v1:172.21.15.55:6789/0 conn(0x7fd51c143d10 legacy=0x7fd51c070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.313+0000 7fd518ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:53:01.322 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.313+0000 7fd518ff9640 1 monclient: found mon.b 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.313+0000 7fd518ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd518ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c1548d0 con 0x7fd51c0b1f60 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd518ff9640 10 monclient: _finish_auth 0 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd518ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd518ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd522990640 5 monclient: authenticate success, global_id 4597 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd522990640 10 monclient: _renew_subs 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd522990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51c132f30 con 0x7fd51c0b1f60 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd522990640 10 client.4597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd522990640 10 monclient: _renew_subs 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd522990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51c155060 con 0x7fd51c0b1f60 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd522990640 1 librados: init done 2024-03-29T11:53:01.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd518ff9640 10 client.4597.objecter ms_handle_connect 0x7fd51c143d10 2024-03-29T11:53:01.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.314+0000 7fd518ff9640 10 client.4597.objecter resend_mon_ops 2024-03-29T11:53:01.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.315+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd50c001c80 con 0x7fd51c0b1f60 2024-03-29T11:53:01.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.315+0000 7fd518ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:01.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.315+0000 7fd51a7fc640 4 set_mon_vals no callback set 2024-03-29T11:53:01.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.316+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7fd50c003a40 con 0x7fd51c0b1f60 2024-03-29T11:53:01.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.316+0000 7fd518ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:01.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.316+0000 7fd518ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:53:01.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.316+0000 7fd518ff9640 10 monclient: dump: 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:53:01.325 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:01.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.317+0000 7fd522990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.317+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7fd50c003c60 con 0x7fd51c0b1f60 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.318+0000 7fd522990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.318+0000 7fd522990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.318+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc005080 con 0x7fd51c0b1f60 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.318+0000 7fd51b7fe640 10 monclient: get_auth_request con 0x7fd50806db60 auth_method 0 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.318+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2667449288 0 0) 0x7fd50c084b70 con 0x7fd51c0b1f60 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.318+0000 7fd518ff9640 10 client.4597.objecter ms_dispatch 0x7fd51c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.318+0000 7fd518ff9640 3 client.4597.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.318+0000 7fd518ff9640 3 client.4597.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.319+0000 7fd518ff9640 20 client.4597.objecter dump_active .. 0 homeless 2024-03-29T11:53:01.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.320+0000 7fd518ff9640 10 client.4597.objecter ms_handle_connect 0x7fd50806db60 2024-03-29T11:53:01.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.325+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd50c050450 con 0x7fd51c0b1f60 2024-03-29T11:53:01.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.325+0000 7fd518ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:01.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.325+0000 7fd518ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:01.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.654+0000 7fd522990640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:01.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.654+0000 7fd522990640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:01.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.654+0000 7fd522990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-29T11:53:01.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.654+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4fc005330 con 0x7fd51c0b1f60 2024-03-29T11:53:01.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.655+0000 7fd518ff9640 1 -- 172.21.15.55:0/298572894 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 824645091) 0x7fd50c052380 con 0x7fd51c0b1f60 2024-03-29T11:53:01.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.655+0000 7fd518ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:01.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.655+0000 7fd518ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:01.667 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:53:01.668 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:52:14.925502+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:52:14.925502+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:01.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.660+0000 7fd522990640 10 librados: watch_flush enter 2024-03-29T11:53:01.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.661+0000 7fd522990640 10 librados: watch_flush exit 2024-03-29T11:53:01.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.661+0000 7fd522990640 20 client.4597.objecter shutdown clearing up homeless session... 2024-03-29T11:53:01.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.661+0000 7fd522990640 10 client.4597.objecter successfully canceled tick 2024-03-29T11:53:01.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.662+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 >> v1:172.21.15.171:6816/1463197163 conn(0x7fd50806db60 legacy=0x7fd508070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:01.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.662+0000 7fd522990640 10 monclient: shutdown 2024-03-29T11:53:01.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.662+0000 7fd522990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:01.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.662+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 >> v1:172.21.15.171:6789/0 conn(0x7fd51c0b1f60 legacy=0x7fd51c1dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:01.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.662+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 shutdown_connections 2024-03-29T11:53:01.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.662+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 >> 172.21.15.55:0/298572894 conn(0x7fd51c066a00 msgr2=0x7fd51c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:01.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.664+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 shutdown_connections 2024-03-29T11:53:01.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.664+0000 7fd522990640 1 -- 172.21.15.55:0/298572894 wait complete. 2024-03-29T11:53:01.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:01.667+0000 7fd522990640 1 librados: shutdown 2024-03-29T11:53:01.691 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:08.499 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:08.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.646+0000 7f058962b640 10 monclient: get_monmap_and_config 2024-03-29T11:53:08.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.646+0000 7f058962b640 10 monclient: build_initial_monmap 2024-03-29T11:53:08.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.646+0000 7f058962b640 10 monclient: monmap: 2024-03-29T11:53:08.655 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:53:08.655 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:08.655 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:53:08.646941+0000 2024-03-29T11:53:08.655 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:53:08.646941+0000 2024-03-29T11:53:08.656 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:08.656 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:08.656 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:53:08.656 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-29T11:53:08.656 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-29T11:53:08.656 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:08.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.648+0000 7f058962b640 1 Processor -- start 2024-03-29T11:53:08.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.648+0000 7f058962b640 1 -- start start 2024-03-29T11:53:08.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.648+0000 7f058962b640 10 monclient: init 2024-03-29T11:53:08.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:08.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:53:08.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 10 monclient(hunting): picked mon.noname-a con 0x7f057c0e2d10 addr v1:172.21.15.55:6789/0 2024-03-29T11:53:08.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 10 monclient(hunting): picked mon.noname-c con 0x7f057c02b090 addr v1:172.21.15.171:6789/0 2024-03-29T11:53:08.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 10 monclient(hunting): picked mon.noname-b con 0x7f057c02c7f0 addr v1:172.21.15.55:6790/0 2024-03-29T11:53:08.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c02ac40 con 0x7f057c0e2d10 2024-03-29T11:53:08.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c02a430 con 0x7f057c02b090 2024-03-29T11:53:08.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c02a5c0 con 0x7f057c02c7f0 2024-03-29T11:53:08.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:08.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.649+0000 7f058962b640 10 monclient(hunting): authenticate will time out at 1601.927877s 2024-03-29T11:53:08.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.653+0000 7f0583fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f057c0e2d10 0x7f057c0e8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42207/0 (socket says 172.21.15.55:42207) 2024-03-29T11:53:08.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.653+0000 7f0583fff640 1 -- 172.21.15.55:0/3989963477 learned_addr learned my addr 172.21.15.55:0/3989963477 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:53:08.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.654+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710698755 0 0) 0x7f057c02ac40 con 0x7f057c0e2d10 2024-03-29T11:53:08.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.654+0000 7f0582ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:53:08.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.654+0000 7f0582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:08.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.654+0000 7f0582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:08.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.654+0000 7f0582ffd640 10 monclient(hunting): my global_id is 4623 2024-03-29T11:53:08.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.654+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0574003520 con 0x7f057c0e2d10 2024-03-29T11:53:08.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.655+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488412280 0 0) 0x7f057c02a430 con 0x7f057c02b090 2024-03-29T11:53:08.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.655+0000 7f0582ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.655+0000 7f0582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.655+0000 7f0582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.655+0000 7f0582ffd640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.655+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f057c02ac40 con 0x7f057c02b090 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671880035 0 0) 0x7f0574003520 con 0x7f057c0e2d10 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 >> v1:172.21.15.55:6790/0 conn(0x7f057c02c7f0 legacy=0x7f057c0e9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 >> v1:172.21.15.171:6789/0 conn(0x7f057c02b090 legacy=0x7f057c029c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 1 monclient: found mon.noname-a 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c006940 con 0x7f057c0e2d10 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 10 monclient: _finish_auth 0 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:53:08.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f058962b640 5 monclient: authenticate success, global_id 4623 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f058962b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0578001690 con 0x7f057c0e2d10 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f0582ffd640 10 monclient: dump: 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:53:08.668 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.656+0000 7f058962b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.658+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05780018b0 con 0x7f057c0e2d10 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.658+0000 7f0582ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.658+0000 7f058962b640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.658+0000 7f058962b640 4 set_mon_vals no callback set 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.659+0000 7f0582ffd640 1 -- 172.21.15.55:0/3989963477 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0578001c40 con 0x7f057c0e2d10 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.659+0000 7f0582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.659+0000 7f0582ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.659+0000 7f0582ffd640 10 monclient: dump: 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:08.669 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:08.670 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:53:08.670 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:53:08.670 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:53:08.670 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:08.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.660+0000 7f058962b640 10 monclient: shutdown 2024-03-29T11:53:08.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.660+0000 7f058962b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:08.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.660+0000 7f058962b640 1 -- 172.21.15.55:0/3989963477 >> v1:172.21.15.55:6789/0 conn(0x7f057c0e2d10 legacy=0x7f057c0e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:08.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.663+0000 7f058962b640 1 -- 172.21.15.55:0/3989963477 shutdown_connections 2024-03-29T11:53:08.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.663+0000 7f058962b640 1 -- 172.21.15.55:0/3989963477 >> 172.21.15.55:0/3989963477 conn(0x7f057c0058f0 msgr2=0x7f057c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:08.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.665+0000 7f058962b640 1 -- 172.21.15.55:0/3989963477 shutdown_connections 2024-03-29T11:53:08.673 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.666+0000 7f058962b640 1 -- 172.21.15.55:0/3989963477 wait complete. 2024-03-29T11:53:08.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.667+0000 7f058962b640 10 monclient: build_initial_monmap 2024-03-29T11:53:08.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.667+0000 7f058962b640 10 monclient: monmap: 2024-03-29T11:53:08.674 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-29T11:53:08.674 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.667+0000 7f058962b640 1 librados: starting msgr at 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.667+0000 7f058962b640 1 librados: starting objecter 2024-03-29T11:53:08.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.667+0000 7f058962b640 1 Processor -- start 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.668+0000 7f058962b640 1 -- start start 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.668+0000 7f058962b640 1 librados: setting wanted keys 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.668+0000 7f058962b640 1 librados: calling monclient init 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.668+0000 7f058962b640 10 monclient: init 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 10 monclient(hunting): picked mon.noname-c con 0x7f057c02b090 addr v1:172.21.15.55:6790/0 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 10 monclient(hunting): picked mon.noname-a con 0x7f057c02c7f0 addr v1:172.21.15.55:6789/0 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 10 monclient(hunting): picked mon.noname-b con 0x7f057c0e2d10 addr v1:172.21.15.171:6789/0 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c02f7e0 con 0x7f057c02c7f0 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c02fac0 con 0x7f057c0e2d10 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c02e8a0 con 0x7f057c02b090 2024-03-29T11:53:08.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:08.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.669+0000 7f058962b640 10 monclient(hunting): authenticate will time out at 1601.948002s 2024-03-29T11:53:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.672+0000 7f0583fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f057c02b090 0x7f057c02d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:40693/0 (socket says 172.21.15.55:40693) 2024-03-29T11:53:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.672+0000 7f0583fff640 1 -- 172.21.15.55:0/3355612338 learned_addr learned my addr 172.21.15.55:0/3355612338 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-29T11:53:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.675+0000 7f0580ff9640 10 client.?.objecter ms_handle_connect 0x7f057c02b090 2024-03-29T11:53:08.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.675+0000 7f0580ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.675+0000 7f0580ff9640 10 client.?.objecter ms_handle_connect 0x7f057c02c7f0 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.675+0000 7f0580ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817736684 0 0) 0x7f057c02f7e0 con 0x7f057c02c7f0 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 monclient(hunting): my global_id is 4626 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0568004eb0 con 0x7f057c02c7f0 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869039372 0 0) 0x7f057c02e8a0 con 0x7f057c02b090 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0568006230 con 0x7f057c02b090 2024-03-29T11:53:08.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 client.?.objecter ms_handle_connect 0x7f057c0e2d10 2024-03-29T11:53:08.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.676+0000 7f0580ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.679+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2522874456 0 0) 0x7f0568006230 con 0x7f057c02b090 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f057c02e8a0 con 0x7f057c02b090 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1301614568 0 0) 0x7f0568004eb0 con 0x7f057c02c7f0 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0568006230 con 0x7f057c02c7f0 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862037996 0 0) 0x7f057c02fac0 con 0x7f057c0e2d10 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): my global_id is 4603 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0568004eb0 con 0x7f057c0e2d10 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0578001bc0 con 0x7f057c02b090 2024-03-29T11:53:08.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): dump: 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f05700018f0 con 0x7f057c02c7f0 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:08.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient(hunting): dump: 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 664028378 0 0) 0x7f057c02e8a0 con 0x7f057c02b090 2024-03-29T11:53:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 >> v1:172.21.15.171:6789/0 conn(0x7f057c0e2d10 legacy=0x7f057c0e44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 >> v1:172.21.15.55:6789/0 conn(0x7f057c02c7f0 legacy=0x7f057c031220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 monclient: found mon.c 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c02e350 con 0x7f057c02b090 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient: _finish_auth 0 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f0580ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f058962b640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:53:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f058962b640 10 monclient: _renew_subs 2024-03-29T11:53:08.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f058962b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:53:08.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c02e600 con 0x7f057c02b090 2024-03-29T11:53:08.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f058962b640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:08.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f058962b640 10 monclient: _renew_subs 2024-03-29T11:53:08.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f058962b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:53:08.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c031b10 con 0x7f057c02b090 2024-03-29T11:53:08.691 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.680+0000 7f058962b640 1 librados: init done 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.681+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0578004a90 con 0x7f057c02b090 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.681+0000 7f0580ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.681+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 861449588 0 0) 0x7f0578004da0 con 0x7f057c02b090 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.681+0000 7f0580ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.681+0000 7f0580ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.681+0000 7f0580ff9640 10 monclient: dump: 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:fsid 4d42039a-57d6-490b-b589-aba66145591c 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-29T11:41:28.509848+0000 2024-03-29T11:53:08.692 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:08.693 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-29T11:53:08.693 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-29T11:53:08.693 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-29T11:53:08.693 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-29T11:53:08.693 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-29T11:53:08.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.681+0000 7f058962b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:08.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.682+0000 7f058962b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:08.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.682+0000 7f058962b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:53:08.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.682+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c02b620 con 0x7f057c02b090 2024-03-29T11:53:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.683+0000 7f05827fc640 4 set_mon_vals no callback set 2024-03-29T11:53:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.686+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3570674676 0 0) 0x7f057801ad70 con 0x7f057c02b090 2024-03-29T11:53:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.687+0000 7f05837fe640 10 monclient: get_auth_request con 0x7f056806e0d0 auth_method 0 2024-03-29T11:53:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.687+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2667449288 0 0) 0x7f0578001d20 con 0x7f057c02b090 2024-03-29T11:53:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.687+0000 7f0580ff9640 10 client.4652.objecter ms_dispatch 0x7f0584000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:53:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.687+0000 7f0580ff9640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:53:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.687+0000 7f0580ff9640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:53:08.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.687+0000 7f0580ff9640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T11:53:08.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.687+0000 7f0580ff9640 10 client.4652.objecter ms_handle_connect 0x7f056806e0d0 2024-03-29T11:53:08.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.697+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0578050b20 con 0x7f057c02b090 2024-03-29T11:53:08.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.697+0000 7f0580ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:08.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:08.697+0000 7f0580ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:09.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.036+0000 7f058962b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:09.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.036+0000 7f058962b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:09.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.036+0000 7f058962b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-29T11:53:09.043 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.036+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f057c031dc0 con 0x7f057c02b090 2024-03-29T11:53:09.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.039+0000 7f0580ff9640 1 -- 172.21.15.55:0/3355612338 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 824645091) 0x7f0578050b20 con 0x7f057c02b090 2024-03-29T11:53:09.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.039+0000 7f0580ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:09.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.039+0000 7f0580ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:09.047 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-29T11:53:09.047 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"4d42039a-57d6-490b-b589-aba66145591c","created":"2024-03-29T11:42:10.030197+0000","modified":"2024-03-29T11:52:14.925502+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:52:14.925502+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64e3cee1-493f-490b-8c64-884ddab30d9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b8c09668-0d5e-4a37-bac5-dcfaaa0e3677","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"49182b31-63d6-416d-a58c-71143c8d99f2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07800f7f-877c-404b-9c2b-d1de99c44a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69fce8b9-b724-4494-be9c-77fe3b3a509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c44f523-b8c3-480d-8239-3170a17ba3e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"546bd470-ae95-4400-b122-5629405751e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c466d742-1d0d-4f6f-83c1-9572380d0809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 10 librados: watch_flush enter 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 10 librados: watch_flush exit 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 10 client.4652.objecter successfully canceled tick 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 >> v1:172.21.15.171:6816/1463197163 conn(0x7f056806e0d0 legacy=0x7f0568070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 10 monclient: shutdown 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 >> v1:172.21.15.55:6790/0 conn(0x7f057c02b090 legacy=0x7f057c02d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 shutdown_connections 2024-03-29T11:53:09.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.043+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 >> 172.21.15.55:0/3355612338 conn(0x7f057c0058f0 msgr2=0x7f057c0e9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:09.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.044+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 shutdown_connections 2024-03-29T11:53:09.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.044+0000 7f058962b640 1 -- 172.21.15.55:0/3355612338 wait complete. 2024-03-29T11:53:09.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-29T11:53:09.045+0000 7f058962b640 1 librados: shutdown 2024-03-29T11:53:09.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:09.069 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:53:09.071 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:53:09.071 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:53:09.071 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.114 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:53:09.115 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:53:09.115 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.160 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:53:09.160 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:53:09.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.193 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:53:09.193 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:53:09.194 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.263 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:53:09.263 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:53:09.263 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.306 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:53:09.306 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:53:09.306 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.337 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:53:09.338 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:53:09.338 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.408 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:53:09.408 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:53:09.408 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.493 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:53:09.493 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:53:09.494 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:53:09.494 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.517 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:53:09.517 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:53:09.518 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.542 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:53:09.542 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:53:09.542 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:53:09.542 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.570 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:53:09.570 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:53:09.570 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.624 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:53:09.624 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:53:09.624 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:53:09.635 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:53:09.635 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:53:09.636 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:53:09.695 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:53:09.695 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:53:12.391 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:53:12.391 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:53:12.487 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:53:12.487 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:53:12.544 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:53:12.544 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:53:12.600 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:53:12.601 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:53:16.030 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:53:16.030 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:53:16.123 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:53:16.124 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:53:16.181 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:53:16.181 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:53:16.267 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:53:16.268 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/data/mon.a.tgz 2024-03-29T11:53:16.269 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-29T11:53:16.286 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.n981VmxTud 2024-03-29T11:53:16.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.n981VmxTud 2024-03-29T11:53:16.402 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.n981VmxTud 2024-03-29T11:53:16.490 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.n981VmxTud is 211KB 2024-03-29T11:53:16.585 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.n981VmxTud 2024-03-29T11:53:16.600 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/data/mon.c.tgz 2024-03-29T11:53:16.600 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-29T11:53:16.654 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.iXOZ2wILpw 2024-03-29T11:53:16.655 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.iXOZ2wILpw 2024-03-29T11:53:16.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.iXOZ2wILpw 2024-03-29T11:53:16.841 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.iXOZ2wILpw is 211KB 2024-03-29T11:53:16.934 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.iXOZ2wILpw 2024-03-29T11:53:16.948 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/data/mon.b.tgz 2024-03-29T11:53:16.948 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-03-29T11:53:16.966 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.9Yw2MvgK9F 2024-03-29T11:53:16.966 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.9Yw2MvgK9F 2024-03-29T11:53:17.084 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.9Yw2MvgK9F 2024-03-29T11:53:17.171 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.9Yw2MvgK9F is 211KB 2024-03-29T11:53:17.265 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.9Yw2MvgK9F 2024-03-29T11:53:17.280 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:53:17.280 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:53:17.282 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:53:17.362 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:53:17.402 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:53:17.402 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:53:17.404 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/remote/smithi055/crash 2024-03-29T11:53:17.405 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:53:17.432 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/remote/smithi171/crash 2024-03-29T11:53:17.433 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:53:17.468 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:53:17.469 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:53:17.476 DEBUG:teuthology.orchestra.run.smithi171:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:53:17.537 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:53:17.537 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39700.log 2024-03-29T11:53:17.537 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39071.log 2024-03-29T11:53:17.537 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39345.log 2024-03-29T11:53:17.537 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39071.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39700.log.gz 2024-03-29T11:53:17.537 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34804.log 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38250.log 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39071.log.gz 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39305.log 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34804.log.gz 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39462.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38250.log.gz -5 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39504.log 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39305.log.gz 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-03-29T11:53:17.538 INFO:teuthology.orchestra.run.smithi055.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38837.log 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39504.log.gz 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40246.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38837.log: gzip -5 --verbose -- 86.5% /var/log/ceph/ceph-client.admin.40754.log 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38993.log 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38837.log.gz 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39895.log 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40871.log 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38993.log: /var/log/ceph/ceph-client.admin.39895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-03-29T11:53:17.539 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38993.log.gz 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39895.log.gz 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40871.log.gz 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38486.log 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41066.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38759.log 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38759.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38486.log.gz 2024-03-29T11:53:17.540 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40012.log 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38367.log 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38759.log.gz 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40012.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38367.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40012.log.gz 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39661.log 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38915.log 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38367.log.gz 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38211.log 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38915.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40051.log 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39661.log.gz 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38211.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38915.log.gz 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38408.log 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2024-03-29T11:53:17.541 INFO:teuthology.orchestra.run.smithi055.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38211.log.gz 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41417.log 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40051.log.gz 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38408.log.gz 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41378.log 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40832.log 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41417.log.gz 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40558.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41378.log.gz 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-03-29T11:53:17.542 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40832.log.gz 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39384.log: gzip -5 --verbose -- 86.2% /var/log/ceph/ceph-client.admin.40480.log 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41456.log 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41261.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38289.log 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41456.log: 86.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.40480.log.gz -5 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38447.log 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38289.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41456.log.gz 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38289.log.gz 2024-03-29T11:53:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39973.log 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40636.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38447.log.gz 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40715.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39973.log.gz 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41573.log 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40441.log: /var/log/ceph/ceph-client.admin.41573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.3.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-03-29T11:53:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39227.log 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41573.log.gz 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39423.log 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41534.log 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39423.log: gzip 86.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.39227.log.gz --verbose 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr: -- /var/log/ceph/ceph-client.admin.38681.log 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38642.log 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41534.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39423.log.gz 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39778.log 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41534.log.gz 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38642.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38681.log.gz 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41027.log 86.4% 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38642.log.gz 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39778.log: /var/log/ceph/ceph-client.admin.40988.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28706.log 2024-03-29T11:53:17.545 INFO:teuthology.orchestra.run.smithi055.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39778.log.gz 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38172.log 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40910.log 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph.tmp-client.admin.28706.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41027.log.gz 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28706.log.gz 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38172.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39110.log 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38172.log.gz 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39110.log: /var/log/ceph/ceph-client.admin.40285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39543.log 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr: 86.9%/var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-03-29T11:53:17.546 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39110.log.gz 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38525.log 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40090.log 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34907.log 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39543.log.gz 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gzgzip 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40949.log 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40090.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38525.log.gz 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39582.log 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40949.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40090.log.gz 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34907.log.gz 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39582.log: gzip -5 --verbose -- 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40949.log.gz 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr: /var/log/ceph/ceph-client.admin.38133.log 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.0.log: 85.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39266.log 2024-03-29T11:53:17.547 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39582.log.gz 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38876.log 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39188.log 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39266.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38133.log.gz 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40207.log 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr: 86.1%/var/log/ceph/ceph-client.admin.39188.log: -- replaced with /var/log/ceph/ceph-client.admin.39266.log.gz 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38876.log.gz 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39188.log.gz 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38720.log 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40207.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38720.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40207.log.gz 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39149.log 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38720.log.gz 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.2.log: gzip 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2024-03-29T11:53:17.548 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39817.log 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40793.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39149.log.gz 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39817.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39817.log.gz 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38798.log 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr:gzip/var/log/ceph/ceph-client.admin.40129.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39032.log 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz/var/log/ceph/ceph-client.admin.38798.log: 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.38798.log.gz 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41300.log 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34513.log 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39032.log.gz 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41495.log 2024-03-29T11:53:17.549 INFO:teuthology.orchestra.run.smithi055.stderr: 87.5%/var/log/ceph/ceph-client.admin.41300.log: -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34513.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41300.log.gz 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34513.log.gz 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38954.log 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41495.log: /var/log/ceph/ceph-client.admin.38603.log 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39856.log 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41495.log.gz 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5%/var/log/ceph/ceph-client.admin.38603.log: -- replaced with /var/log/ceph/ceph-client.admin.38954.log.gz 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38564.log 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39739.log 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38564.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38603.log.gz 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39856.log.gz 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38328.log 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38564.log.gz 2024-03-29T11:53:17.550 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:53:17.551 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39739.log: /var/log/ceph/ceph-client.admin.38328.log: /var/log/ceph/ceph-mon.c.log: 87.0% 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39739.log.gz 2024-03-29T11:53:17.551 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38328.log.gz 2024-03-29T11:53:17.556 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:53:17.556 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34307.log 2024-03-29T11:53:17.556 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34267.log 2024-03-29T11:53:17.556 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34427.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34467.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34387.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34267.log: /var/log/ceph/ceph-client.admin.34427.log: /var/log/ceph/ceph-client.admin.34467.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34307.log.gz 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:gzip/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34427.log.gz 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34387.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34467.log.gz 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34267.log.gz 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34507.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.7.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34387.log.gz 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34347.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34507.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:53:17.557 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34347.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:53:17.558 INFO:teuthology.orchestra.run.smithi171.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34507.log.gz 2024-03-29T11:53:17.558 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34347.log.gz 2024-03-29T11:53:17.558 INFO:teuthology.orchestra.run.smithi171.stderr: --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:53:17.558 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34227.log 2024-03-29T11:53:17.558 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-client.admin.34227.log: 2024-03-29T11:53:17.558 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34227.log.gz 2024-03-29T11:53:17.613 INFO:teuthology.orchestra.run.smithi171.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:53:29.297 INFO:teuthology.orchestra.run.smithi171.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:53:29.307 INFO:teuthology.orchestra.run.smithi171.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:53:29.385 INFO:teuthology.orchestra.run.smithi171.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:53:29.443 INFO:teuthology.orchestra.run.smithi055.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:53:29.664 INFO:teuthology.orchestra.run.smithi055.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:53:33.387 INFO:teuthology.orchestra.run.smithi171.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:53:33.928 INFO:teuthology.orchestra.run.smithi055.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:53:34.020 INFO:teuthology.orchestra.run.smithi055.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:54:09.890 INFO:teuthology.orchestra.run.smithi055.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:54:12.386 INFO:teuthology.orchestra.run.smithi055.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:54:12.388 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-29T11:54:12.388 INFO:teuthology.orchestra.run.smithi055.stderr:real 0m54.899s 2024-03-29T11:54:12.388 INFO:teuthology.orchestra.run.smithi055.stderr:user 2m38.795s 2024-03-29T11:54:12.388 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m5.189s 2024-03-29T11:54:28.383 INFO:teuthology.orchestra.run.smithi171.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:54:28.385 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-29T11:54:28.385 INFO:teuthology.orchestra.run.smithi171.stderr:real 1m10.857s 2024-03-29T11:54:28.385 INFO:teuthology.orchestra.run.smithi171.stderr:user 1m57.418s 2024-03-29T11:54:28.385 INFO:teuthology.orchestra.run.smithi171.stderr:sys 0m4.497s 2024-03-29T11:54:28.386 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:54:28.386 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/remote/smithi055/log 2024-03-29T11:54:28.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:54:50.679 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/remote/smithi171/log 2024-03-29T11:54:50.680 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:55:08.745 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:55:08.963 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f7644c1e0ab04936b8bef19ffd677257 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:55:08.966 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:55:08.976 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:55:09.019 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:55:09.019 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:55:09.033 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:55:09.133 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:55:09.142 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:55:09.142 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:55:09.145 DEBUG:teuthology.orchestra.run.smithi171:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:55:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:55:09.162 INFO:teuthology.orchestra.run.smithi171.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:55:09.164 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:55:09.165 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-03-29T11:55:09.165 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +361us[ +361us] +/- 79ms 2024-03-29T11:55:09.165 INFO:teuthology.orchestra.run.smithi055.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -124us[ -125us] +/- 74ms 2024-03-29T11:55:09.165 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -180us[ -180us] +/- 74ms 2024-03-29T11:55:09.165 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:55:09.167 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:55:09.167 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-03-29T11:55:09.167 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +325us[ +325us] +/- 79ms 2024-03-29T11:55:09.167 INFO:teuthology.orchestra.run.smithi171.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -123us[ -122us] +/- 74ms 2024-03-29T11:55:09.167 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 -207us[ -206us] +/- 74ms 2024-03-29T11:55:09.167 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:55:09.168 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:55:09.177 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:55:09.177 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:55:09.202 DEBUG:teuthology.orchestra.run.smithi055:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:55:09.277 DEBUG:teuthology.orchestra.run.smithi171:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:55:09.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:55:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:55:09.380 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:55:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:55:09.412 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:55:09.421 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:55:09.452 INFO:teuthology.task.internal:Duration was 1361.033003 seconds 2024-03-29T11:55:09.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:55:09.495 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:55:09.495 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:55:09.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:55:09.541 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:55:09.543 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:55:09.888 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:55:09.888 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi055.front.sepia.ceph.com 2024-03-29T11:55:09.888 DEBUG:teuthology.orchestra.run.smithi055:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:55:09.910 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi171.front.sepia.ceph.com 2024-03-29T11:55:09.910 DEBUG:teuthology.orchestra.run.smithi171:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:55:09.932 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:55:09.932 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:55:09.952 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:55:10.001 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:55:10.001 DEBUG:teuthology.orchestra.run.smithi055:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:55:10.023 DEBUG:teuthology.orchestra.run.smithi171:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:55:10.137 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:55:10.146 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:55:10.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:55:10.173 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:55:10.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:55:10.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:55:10.215 DEBUG:teuthology.orchestra.run.smithi171:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:55:10.238 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2024-03-29T11:55:10.266 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = core 2024-03-29T11:55:10.286 DEBUG:teuthology.orchestra.run.smithi055:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:55:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:55:10.312 DEBUG:teuthology.orchestra.run.smithi171:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:55:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:55:10.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:55:10.350 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:55:10.351 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/remote/smithi055 2024-03-29T11:55:10.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:55:10.391 DEBUG:teuthology.misc:Transferring archived files from smithi171:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733/remote/smithi171 2024-03-29T11:55:10.391 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:55:10.427 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:55:10.427 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:55:10.430 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:55:10.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:55:10.488 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:55:10.489 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:55:10.518 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:55:10.518 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:55:10.521 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:55:10.532 INFO:teuthology.orchestra.run.smithi055.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:55 /home/ubuntu/cephtest 2024-03-29T11:55:10.535 INFO:teuthology.orchestra.run.smithi171.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:55 /home/ubuntu/cephtest 2024-03-29T11:55:10.536 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:55:10.554 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:55:10.629 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:55:10.655 DEBUG:teuthology.nuke:shortname: smithi055 2024-03-29T11:55:10.655 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:55:10.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:22:16.193973', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDn8hr+R//BuBItQCC5dVoSKBKDvgSrKDS+QH9rlJSAYWFy2nL/xzTn7JCmO18nSwnYdEASNcybaXfNAOMWEN/4='} 2024-03-29T11:55:10.684 DEBUG:teuthology.nuke:shortname: smithi171 2024-03-29T11:55:10.684 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:55:10.703 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622733', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:22:16.195403', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOFSxlvW3aKHYhb1nDKtEDx052k4LRqJy3lWvFvDkLSuD/IqfruqCmgITtVgCg5NNInCZA9x6Fe/yzLIJ4Iktqs='} 2024-03-29T11:55:10.735 INFO:teuthology.orchestra.console.smithi055:Power off 2024-03-29T11:55:10.735 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:55:10.750 INFO:teuthology.orchestra.console.smithi171:Power off 2024-03-29T11:55:10.750 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:55:10.759 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-03-29T11:55:10.760 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:55:10.773 DEBUG:teuthology.orchestra.console.smithi171:power off output: Chassis Power Control: Down/Off 2024-03-29T11:55:10.773 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:55:10.782 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-29T11:55:10.794 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-03-29T11:55:14.784 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:55:14.796 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:55:14.895 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-29T11:55:14.906 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-03-29T11:55:18.898 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:55:18.907 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:55:19.009 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-29T11:55:19.019 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-03-29T11:55:19.110 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-03-29T11:55:23.022 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:55:23.132 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is off 2024-03-29T11:55:23.233 INFO:teuthology.orchestra.console.smithi171:Power off completed 2024-03-29T11:55:23.337 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1361.033002614975 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f7644c1e0ab04936b8bef19ffd677257 status: fail success: false 2024-03-29T11:55:23.338 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:55:23.409 INFO:teuthology.run:FAIL